Method and apparatus for determining a direction of interest

ABSTRACT

Methods and apparatus for determining a direction of interest based upon polar coordinates derived from wireless communication between wireless transceivers. A smart device assembly is operative to communicate via multiple antennas with a reference point transceiver. A set of polar coordinates is generated indicating a relative position and angle of the wireless transceiver in relation to the reference position transceiver. A query may be made based upon the relative position and angle of the wireless transceiver in relation to the reference position transceiver. A response to the query may include a human readable interface indicating one or more of: direction of travel, a virtual image based upon location and location and direction, and annotative and pictorial information.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a Continuation Application to Non Provisionalpatent application Ser. No. 16/688,775 filed Nov. 19, 2019, entitledMETHOD AND APPARATUS FOR WIRELESS DETERMINATION OF POSITION ANDORIENTATION OF A SMART DEVICE. The matter Ser. No. 16/688,775 in turnclaimed priority as a Continuation in Part Application to NonProvisional patent application Ser. No. 16/657,660 filed Oct. 18, 2019.The matter Ser. No. 16/688,775 in turn claimed priority as aContinuation in Part Application to Non Provisional patent applicationSer. No. 16/597,271, which claims the benefit of Provisional ApplicationNo. 62/909,061 filed on Oct. 1, 2019. The matter Ser. No. 16/688,775also claims priority as a Continuation in Part to the Non-Provisionalpatent application Ser. No. 16/528,104 filed Jul. 31, 2019, which claimsthe benefit of Provisional Application No. 62/871,499 filed on Jul. 8,2019. The matter Ser. No. 16/688,775 also claimed priority to NonProvisional patent application Ser. No. 16/504,919 as a continuation inpart filed on Jul. 8, 2019. The matter Ser. No. 16/688,775 also claimedpriority as a continuation in part to Non Provisional patent applicationSer. No. 16/503,878 filed Jul. 5, 2019, entitled METHOD AND APPARATUSFOR ENHANCED AUTOMATED WIRELESS ORIENTEERING, which is a Continuation inPart Application to Non Provisional application Ser. No. 16/249,574filed Jan. 16, 2019, and to Non Provisional patent application Ser. No.16/142,275 filed Sep. 26, 2018, entitled METHODS AND APPARATUS FORORIENTEERING as a Continuation in Part Application; which claimspriority to Non Provisional patent application Ser. No. 16/176,002 filedOct. 31, 2018, entitled SYSTEM FOR CONDUCTING A SERVICE CALL WITHORIENTEERING as a Continuation in Part Application, and also to NonProvisional patent application Ser. No. 16/171,593 filed Oct. 26, 2018,entitled SYSTEM FOR HIERARCHICAL ACTIONS BASED UPON MONITORED BUILDINGCONDITIONS as a Continuation in Part Application, and also to NonProvisional patent application Ser. No. 16/165,517, filed Oct. 19, 2018and entitled BUILDING VITAL CONDITIONS MONITORING as a Continuation inPart Application; and to Non Provisional patent application Ser. No.16/161,823, filed Oct. 16, 2018 and entitled BUILDING MODEL WITH CAPTUREOF AS BUILT FEATURES AND EXPERIENTIAL DATA as a Continuation in PartApplication; and to Non Provisional patent application Ser. No.15/887,637, filed Feb. 2, 2018 and entitled BUILDING MODEL WITH CAPTUREOF AS BUILT FEATURES AND EXPERIENTIAL DATA as a Continuation in PartApplication; and to Non Provisional patent application Ser. No.15/703,310, filed Sep. 13, 2017 and entitled BUILDING MODEL WITH VIRTUALCAPTURE OF AS BUILT FEATURES AND OBJECTIVE PERFORMANCE TRACKING as aContinuation in Part Application; and to Non Provisional patentapplication Ser. No. 15/716,133, filed Sep. 26, 2017 and entitledBUILDING MODEL WITH VIRTUAL CAPTURE OF AS BUILT FEATURES AND OBJECTIVEPERFORMANCE TRACKING as a Continuation in Part Application; which claimspriority to Provisional Patent Application Ser. No. 62/712,714, filedJul. 31, 2018 and entitled BUILDING MODEL WITH AUTOMATED WOOD DESTROYINGORGANISM DETECTION AND MODELING; which claims priority to ProvisionalPatent Application Ser. No. 62/462,347, filed Feb. 22, 2017 and entitledVIRTUAL DESIGN, MODELING AND OPERATIONAL MONITORING SYSTEM; which claimspriority to Provisional Patent Application Ser. No. 62/531,955, filedJul. 13, 2017 and entitled BUILDING MODELING WITH VIRTUAL CAPTURE OF ASBUILT FEATURES; which claims priority to Provisional Patent ApplicationSer. No. 62/531,975 filed Jul. 13, 2017 and entitled BUILDINGMAINTENANCE AND UPDATES WITH VIRTUAL CAPTURE OF AS BUILT FEATURES as acontinuation in part application; The matter Ser. No. 16/249,574 alsoclaims priority to Non Provisional application Ser. No. 16/176,002 filedOct. 31, 2018 now U.S. Pat. No. 10,268,782 issued Apr. 23, 2019 as acontinuation in part. The contents of each of which heretoforereferenced applications and patents are relied upon and incorporatedherein by reference.

FIELD OF THE INVENTION

The present invention relates to methods and apparatus for determining alocation and direction of interest based upon multiple wirelesscommunications. Multiple wireless communications between transceiversare used to generate a location and a direction of interest based uponthe location, the location and direction of interest may be referencedin the provision of content via a user interface.

BACKGROUND OF THE INVENTION

It is known for a geospatial position to be ascertained based upontriangulation techniques. Such triangulation techniques may be basedupon artificial location references, such as satellites and/or celltowers. However, calculation of a position is of limited use withoutbeing able to specify a direction of interest.

In addition, traditional methods of using automated design tools, suchas AutoDesk™ have been focused on the generation of a design plan foruse in construction of a facility, such as a processing plant. Anautomated design tool may be advantageous in the specifying of buildingaspects, materials and placement of features. Aspects may includebuilding features, such as walls, ingress/egress, utilities and evenequipment. However, usefulness of the design plan is also limited absenta direction of interest from any given point.

Similarly, while traditional methods of using automated design tools,such as AutoDesk™, have greatly increased the capabilities of virtualmodels of facilities, very little has been done to quantify a deployedperformance of design features, such as equipment layout, capacity,throughout consumables walls, ingress/egress, windows, ceiling designs,textures, building materials, placement of structural beams, utilities,machinery location, machinery type, machinery capacity equipment.Accurate recreation of such design features in the field requires anindication of both location and direction.

More sophisticated design systems include “virtual reality” models.Virtual reality models may include two dimensional and/or threedimensional views from one or more user selected Vantage Points withinthe model of the structure. Virtual reality models also require adesignation of a Vantage Pont and a direction.

SUMMARY OF THE INVENTION

Accordingly, the present invention combines methods and apparatus fordesignating a geospatial location and a direction of interest based uponwireless transmission and/or reception in a manner that provides forgenerating an angle of arrival and/or an angle of transmission. In someembodiments, a time of transmission and time of arrival may also begenerated, as may be time and date of transmission.

A directional line is virtually formed based upon wirelesstransmissions. The directional line will virtually intersect a spaceoccupied by the smart device. In some embodiments, a subset of the linein the form of a ray may be generated with the ray virtuallyintersecting a space occupied by the smart device. Still further, someembodiments may include an origin point of the ray occupying the virtualspace of the smart device. The line and/or the ray may be used togenerate a direction of interest. Content may be provided based upon thelocation and direction of interest.

The details of one or more examples of the invention are set forth inthe accompanying drawings and the description below. The accompanyingdrawings that are incorporated in and constitute a part of thisspecification illustrate several examples of the invention and, togetherwith the description, serve to explain the principles of the invention:other features, objects, and advantages of the invention will beapparent from the description, drawings, and claims herein.

DESCRIPTION OF THE DRAWINGS

The accompanying drawings, that are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention:

FIG. 1A illustrates a block diagram of inter-relating functions includedin automated systems according to the present invention.

FIG. 1B illustrates geolocation aspects that may be used to identify aproperty and corresponding data and predictions.

FIG. 1C illustrates a block diagram of ongoing data capture via SmartDevices and Sensors and support for predictive modeling based upon thesmart data capture.

FIG. 1D illustrates an exemplary Facility layout with various equipmentdelineated in a top-down representation according to some embodiments ofthe present invention.

FIG. 1E illustrates a diagram of a user and directional image data.

FIG. 2 illustrates a block diagram of an Augmented Virtual Modelingsystem.

FIGS. 3A-3F, are illustrations of exemplary aspects of collecting anddisplaying data of a Processing Facility generated during constructionof the Processing Facility.

FIG. 3G illustrates an exemplary key component of the model system, witha Performance monitor providing data via a communication system to themodel system.

FIG. 3H illustrates an exemplary virtual reality display in according tothe present invention.

FIGS. 4A, 4B, and 4C illustrate exemplary method flow diagrams withsteps relating to processes of the present invention.

FIGS. 5-5A illustrate location and positioning transceivers used forlocation determination.

FIG. 6 illustrates apparatus that may be used to implement aspects ofthe present invention including executable software.

FIG. 7 illustrates an exemplary mobile smart device that may be used toimplement aspects of the present invention including executablesoftware.

FIG. 8 illustrates method steps that may be implemented according tosome aspects of the present invention.

FIGS. 9A-D illustrates views of an AVM via a wearable eye displayaccording to some aspects of the present invention.

FIGS. 10A-C illustrates viewing areas of an AVM according to someaspects of the present invention.

FIGS. 11A-C illustrates vertical changes in an AVM viewable areaaccording to some aspects of the present invention.

FIG. 12 illustrates designation of a direction according to some aspectsof the present invention.

FIGS. 13, 13A-13D illustrate a device and vectors according to variousembodiments of the present invention.

FIG. 14 illustrates a vehicle acting as platform 1400 for supportingwireless position devices.

FIGS. 15A-15C illustrate movement of a smart device to generate avector.

FIGS. 16 and 16A illustrate method steps that may be executed inpracticing some embodiments of the present invention.

FIGS. 17A-B illustrates method steps that may be implemented in someembodiments of the present disclosure.

FIG. 18A-18B illustrates a defined area with Transceivers.

FIG. 18C-18E illustrate devices that may include a Transceiver.

FIG. 19A illustrates wireless communication, including directionalTransceivers.

FIG. 19B illustrates an apparatus with Transceivers and generation of avector.

FIG. 19C illustrates an exemplary apparatus for attaching an exemplaryaccelerometer to a component of a Structure.

FIGS. 20 and 20A-20C illustrate method steps that may be executed inpracticing some embodiments of the present invention.

FIG. 21 illustrates an exemplary method steps for generation of a userinterface.

FIGS. 22A-B illustrates method steps that may be executed in practicingsome embodiments of the present invention.

FIGS. 23A-C illustrate aspects of headset displays with locationdevices.

FIG. 24A illustrates an exemplary interior map with directions basedupon the AVM.

FIG. 24B illustrates an exemplary embodiment of heads-up display for anagent.

FIG. 25A illustrates use of an AV Headgear in concert with a designationof direction.

FIG. 25B illustrates an oriented Headgear in a use mode.

FIG. 25C illustrates an oriented Headgear in a use mode displayingstored information.

FIG. 25D illustrates a Headgear equipped with location viewingstereoscopic cameras in an interactive use mode to establish orientationby moving in an orienteering direction.

FIG. 25E illustrates a Headgear equipped with location viewingstereoscopic cameras in a stereographic imaging mode with a stereoscopicdisplay.

FIG. 25F illustrates a Headgear equipped with location viewingstereoscopic cameras in an interactive use mode to establish orientationby pointing in an orienteering direction while wearing a GPS-equippeddevice.

FIG. 25G illustrates a Headgear equipped with location viewingstereoscopic cameras in an operational mode displaying historicinformation with current view inset.

FIG. 25H illustrates an oriented Headgear equipped with location viewingstereoscopic cameras acquiring current picture data of a viewingdirection with historic view inset.

FIG. 25I illustrates a Headgear equipped with location viewingstereoscopic cameras in an interactive use mode to record panoramicpicture data to update the status record.

FIG. 25J illustrates a Headgear equipped with a handheld camera in aninteractive use mode to record panoramic picture data.

FIGS. 26A-C illustrate an exemplary embodiment of method steps.

FIG. 27A illustrates an exemplary interior map with directions basedupon the AVM.

FIG. 27B illustrates an exemplary embodiment of heads-up display for aservice technician.

FIG. 28A-28E illustrate exemplary diagrams of transceivers and definedareas.

FIG. 29A illustrates an exemplary smart device using GPS data todetermine a coarse-grain location.

FIG. 29B illustrates an exemplary smart device displaying room-levellocation information with reference to WiFi transmitters.

FIG. 29C illustrates an exemplary smart device displaying fine-grainlocation information using Bluetooth transmitters.

FIGS. 30A-30C illustrate various embodiments of antenna arrays.

FIGS. 30D-30G illustrate various embodiments of antenna arrays attachedto smart devices.

FIG. 31A illustrates an exemplary determination of angles of arrival anddeparture using an exemplary antenna array.

FIG. 31B-31C illustrates exemplary determinations of calculateddirection of orientation of a smart device including multiple antennaarrays.

DETAILED DESCRIPTION

The present invention provides for smart structure such as smartinfrastructures with active components that continuously monitor andtransmit a current condition of the infrastructure according to a timerelated index. It includes methods and apparatus for construction,deployment and maintenance of an infrastructure with IntelligentAutomation engaged in Structural Messaging. In some embodiments,Intelligent Automation is combined with machine generated determinationof a location and a direction of interest which may be referenced in theprovision of content of a user interface.

Various embodiments include methods and apparatus for construction,deployment and maintenance of a Infrastructure with IntelligentAutomation (device, system, machine or equipment item) engaged inlogical processes and Structural Messaging to communicate conditionswithin or proximate to the Structure. Structural Messaging includeslogical communications generated by the IA (such as a sensor or machine)incorporated into, affixed to or operated within or proximate to aStructure.

In general, various embodiments of the present invention enable aninfrastructure, such as a building or infrastructure, to be active asopposed to the former passive state. The active state enables thestructure to generate data descriptive of one or more of: a conditionwithin a structure; a condition proximate to the structure; and an eventexperienced by the structure; and in some embodiments an active statestructure is enabled to execute an action via automation based upon aStructural Message. The action based upon a Structural Message may beexecuted independent of a user intervention, or based upon approval of auser, such as via an app on a smart device.

The present invention provides automated apparatus and methods forgenerating improved Augmented Virtual Models (sometimes referred toherein as an “AVM”) of a infrastructure; the improved AVMs are capableof calculating a likelihood of achieving stated Performance Levelspecified by a user and/or a deployment such as public transportation.In addition, the improved model may be operative to generate targetPerformance Metrics based upon As Built and Experiential Data.

The Augmented Virtual Model of the property may include a conceptualmodel and progress through one or more of: a) a design stage; b) a buildstage; c) a Deployment stage; d) a service stage; e) a modificationstage; and f) a dispensing stage. As discussed more fully herein, an AVMaccording to the present invention include original design data matchedto As Built data captured via highly accurate geolocation, direction andelevation determination. As Built data is matched with a time and dateof data acquisition and presented in two dimensional (2D) and threedimensional (3D) visual representations of the property. The augmentedmodels additionally include data relating to features specified in aproperty design and data collected during building, Deployment,maintenance and modifications to the property. In some embodiments, afourth dimension of time may also be included.

An Augmented Virtual Model includes a three or four dimensional model ina virtual environment that exists parallel to physical embodimentsmodeled in the Augmented Virtual Model. Details of one or more physicalstructures and other features within a real estate parcel are generatedand quantified and represented in the Augmented Virtual Model. TheAugmented Virtual Model exists in parallel to a physical structure inthat the AVM includes virtual representations of physical structures andadditionally receives and aggregates data relevant to the structuresover time. The aggregation of data may be one or more of: a) accordingto an episode (i.e. onsite inspection, repair, improvement etc.); b)periodic; and c) in real time (without built in delay).

The experience of the physical infrastructure is duplicated in thevirtual Augmented Virtual Model. The Augmented Virtual Model maycommence via an electronic model generated via traditional CAD softwareor other design type software. In addition, the AVM may be based uponvalues for variables, including one or more of: usage of aninfrastructure; usage of components within the infrastructure;environmental factors encountered during a build stage or Deploymentstage; and metrics related to Performance of the infrastructure. Themetrics may be determined, for example, via measurements performed bySensors located in and proximate to infrastructures located on theproperty.

In another aspect, an Augmented Virtual Model may be accessed inrelation to modeling achievement of a stated Performance Level. Accuratecapture of As Built Features and aggregated data of similar bridges,roadways, dams, power generation units, buildings, equipment types,machinery and usage profiles assist in one or more of: predictingPerformance Level, Yield, Quality, Volume of Production, selectingappropriate technicians to deploy to a service call; providing correctconsumables and replacement parts, scheduling a preventativemaintenance; scheduling building, equipment and/or machinery upgrades;matching a building, equipment and machinery combination of a particulartype of Deployment; providing on site guidance during the Service Call;providing documentation relevant to the building, equipment andmachinery; providing access to remote experts that guide onsitetechnicians.

In some embodiments, a technical library specific to a particularproperty and location within the property may be maintained for eachproperty and made accessible to an onsite technician and/or remoteexpert. The library may include, but is not limited to: structure,equipment/machinery manuals; repair bulletins, and repair/maintenance.Appropriate how to videos may also be made available based upon an AVMwith As Built and Experiential Data.

In another aspect, a parts ordering function may be included in theAugmented Virtual Model. Augmented parts ordering may allow a technicianto view an ordered part and view a virtual demonstration of the part inuse and procedures for replacing the part.

Aspects of the Augmented Virtual Model may be presented via a userinterface that may display on a tablet or other flat screen, or in someembodiments be presented in a virtual reality environment, such as via avirtual reality headset.

The present invention additionally provides for an Augmented VirtualModel to forecast Future Performance of a property based upon the valuesof variables included in data aggregated during the design, build andDeployment of the property sometimes referred to herein as: a) DesignFeatures; b) As Built data; and c) as Deployed data.

The improved modeling system incorporates “As Built” data into theimproved design model. Subsequently, an onsite or remote technician mayaccess the As Built data to facilitate. The As Built data is generatedand/or captured via highly accurate geolocation, direction and elevationdetermination. Based upon the geolocation, direction and elevationdetermination, As Built data is incorporated into a design model at aprecise location within the AVM. In some embodiments, a time and date ofdata acquisition may be associated with updates to aspects of theimproved AVM such that a chronology of changes exists within the AVM.

Original design aspects and updated design aspects may be presented intwo dimensional (2D) and three dimensional (3D) visual representationsof the property. The present invention provides for systematic updatesto As Built data during a Deployment of the property. Updated data mayverify and/or correct previously included data and also be used tomemorialize modifications made during a Service Call or modification toa property.

Some exemplary embodiments may include updates to an AVM that include,one or more of: quantifying a make and model of equipment and machineryon site; time and date notation of change in location specific data;Model accessed and/or updated according to XYZ and distance data; XYdata may include high level location designation within the streetaddress via triangulation (i.e. such as a street address) and highlyspecific position designation (i.e. particular room and wall);combination of two types of position data; GPS, Differential GPS;references used during triangulation; aggregate data across multiplestructures for reference; designs that perform well; designs that fail;popularity of various aspects; access to and/or generation of, multipleAugmented Virtual Models; original and modified model versions; indexaccording to date/time stamp; index according to feature; indexaccording to popularity; index according to cost; index according toUser specific query; plumbing; electrical; HVAC; chemical, raw material,structural; access areas (i.e. crawl spaces, attics); periodic data andposition capture with camera/Sensor attached to a fixed position; andduring one or more of: repair/maintenance/updates.

Accordingly, actual “As Built” imagery and location data areincorporated into the design model to accurately indicate a location andtype of feature included in a structure, and provide “pictures” or othercaptured data. Exemplary data may include As Built locations ofstructural components (beams, headers, doorways, windows, rafters etc.);HVAC, electrical, plumbing, machinery, equipment, etc. A virtual realitymodel may additionally include virtual operation of machinery andequipment and use of a Processing Facility based upon aggregated datafrom the structure, as well as annotations and technical specificationsrelating to features included in the As Built model of a ProcessingFacility identified by time, date, geolocation and direction.

In some embodiments, an initial digital model may be generated accordingto known practices in the industry. However, unlike previously knownpractices, the present invention associates an initial digital modelwith a unique identifier that is logically linked to a geolocation andone or both of date and time designation, and provides updates to theoriginal model based upon data captured at the geolocation during arecorded timeframe. In this manner, a Virtual Reality Simulation isgenerated that logically links a digital model to a specific geographiclocation and actual As Built data at the specific geographic location.The updated model may be virtually accessed from multiple locations suchas a field office, onsite, a technical expert, a financial institution,or other interested party.

In some preferred embodiments, the geographic location will be providedwith accurately placed location reference points. The location referencepoints may be accessed during activities involved in a Service Call onthe property, such as a repair or upgrade to a structure or otherstructures included within a property parcel surrounding the structure.Accuracy of the reference points may or may not be associated withlocation relevance beyond the property, however they do maintainaccuracy within the property.

Preferred embodiments may also include reference points accuratelyplaced within a structure Processing Facility located on the property.As further discussed below, the reference points may include, by way ofnon-limiting example, a wireless transmission data transmitter operativeto transmit an identifier and location data; a visual identifier, suchas a hash code, bar code, color code or the like; an infraredtransmitter; a reflective surface, such as a mirror; or other meanscapable of providing a reference point to be utilized in a triangulationprocess that calculates a precise location within the structure or otherstructure.

Highly accurate location position may be determined via automatedapparatus and multiple levels of increasingly accurate locationdetermination. A first level may include use of a GPS device providing areading to first identify a property. A second level may use positiontransmitters located within, or proximate to, the property to executetriangulation processes in view of on-site location references. A GPSlocation may additionally be associated with a high level generaldescription of a property, such as, one or more of: an address, a unitnumber, a lot number, a taxmap number, a county designation, Plattenumber or other designator. On-site location references may include oneor more of: near field radio communication beacons at known Cartesianposition reference points; line of sight with physical referencemarkers; coded via ID such as bar code, hash code, and alphanumeric orother identifier. In some embodiments, triangulation may calculate aposition within a boundary created by the reference points, whichposition is accurate on the order of millimeters. In some embodiments,Differential GPS may be used to accurately determine a location of aSmart Device with a sub centimeter accuracy. In addition to a positiondetermination, such as latitude and longitude, or other CartesianCoordinate (which may sometimes be indicated as an “X and Y” coordinate)or GPS coordinate, the present invention provides for a direction(sometimes referred to herein as a “Z” direction and elevation) of afeature for which As Built data is captured and imported into the AVM.

In addition to a position determination, such as latitude and longitude,or other Cartesian Coordinate (which may sometimes be indicated as an “Xand Y” coordinate) or GPS coordinate, the present invention provides fora direction (sometimes referred to herein as a “Z” direction andelevation) of a feature for which As Built data is captured and importedinto the AVM.

According to the present invention, a direction dimension may be basedupon a movement of a device. For example, a device with a controller andan accelerometer, such as mobile Smart Device, may include a userdisplay that allows a direction to be indicated by movement of thedevice from a determined location acting as a base position towards anAs Built feature in an extended position. In some implementations, theSmart Device may first determine a first position based upontriangulation with the reference points and a second position (extendedposition) also based upon triangulation with the reference points. Theprocess of determination of a position based upon triangulation with thereference points may be accomplished, for example via executablesoftware interacting with the controller in the Smart Device, such as,for example via running an app on the Smart Device.

In combination with, or in place of directional movement of a deviceutilized to quantify a direction of interest to a user, some embodimentsmay include an electronic and/or magnetic directional indicator that maybe aligned by a user in a direction of interest. Alignment may include,for example, pointing a specified side of a device, or pointing an arrowor other symbol displayed upon a user interface on the device towards adirection of interest.

In a similar fashion, triangulation may be utilized to determine arelative elevation of the Smart Device as compared to a referenceelevation of the reference points.

It should be noted that although a Smart Device is generally operated bya human user, some embodiments of the present invention include acontroller, accelerometer, data storage medium, Image Capture Device,such as a Charge Coupled Device (“CCD”) capture device and/or aninfrared capture device being available in a handheld or unmannedvehicle.

An unmanned vehicle may include for example, an unmanned aerial vehicle(“UAV”) or ground level unit, such as a unit with wheels or tracks formobility and a radio control unit for communication.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or a threedimensional and 4 dimensional (over time) aspect to the captured data.In some implementations, UAV position will be contained within aperimeter and the perimeter will have multiple reference points to helpeach UAV (or other unmanned vehicle) determine a position in relation tostatic features of a building within which it is operating and also inrelation to other unmanned vehicles. Still other aspects includeunmanned vehicles that may not only capture data but also function toperform a task, such as paint a wall, drill a hole, cut along a definedpath, or other function. As stated throughout this disclosure, thecaptured data may be incorporated into the virtual model of a ProcessingFacility.

In another aspect, captured data may be compared to a library of storeddata using image recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Still other aspects mayinclude the use of a compass incorporated into a Smart Device.

In still other implementations, a line of sight from a Smart Device,whether user operated or deployed in an unmanned vehicle, may be used toalign the Smart Device with physical reference markers and therebydetermine an XY position as well as a Z position. Electronic altitudemeasurement may also be used in place of, or to supplement, a knownaltitude of a nearby reference point. This may be particularly useful inthe case of availability of only a single reference point.

Reference points may be coded via identifiers, such as a UUID(Universally Unique Identifier), or other identification vehicle. Visualidentifiers may include a bar code, hash tag, Alphanumeric or othersymbol. Three dimensional markers may also be utilized.

By way of non-limiting example, on site data capture may includedesignation of an XYZ reference position and one or more of: imagecapture; infra-red capture; Temperature; Humidity; Airflow;Pressure/tension; Electromagnetic reading; Radiation reading; Soundreadings (i.e. level of noise, sound pattern to ascertain equipmentrunning and/or state of disrepair), and other vibration or Sensorreadings (such as an accelerometer or transducer).

In some embodiments, vibration data may be used to profile use of thebuilding and/or equipment and machinery associated with the building.For example, vibration detection may be used to determine a machineoperation, including automated determination between proper operation ofa piece of equipment and/or machinery and faulty operation of theequipment and/or machinery. Accelerometers may first quantify facilityoperations and production speed and/or capacity during operations.Accelerometers may also detect less than optimal performance ofequipment and/or machinery. In some embodiments. AI may be used toanalyze and predict proper operation and/or equipment/machinery failurebased upon input factors, including vibration patterns captured.Vibrations may include a “signature” based upon machine type andlocation within a structure human related activity, such as, by way ofnon-limiting example: machine and foot traffic, physical activities,machine operations, machine failure, raised voices, alarms and alerts,loud music, running, dancing and the like, as well as a number ofmachines and/or people in the building and a calculated weight andmobility of the people.

Vibration readings may also be used to quantify operation of machineryand equipment associated with the building, such as HVAC, circulatorsand water pumps. Vibration data may be analyzed to generate profiles forproperly running equipment and equipment that may be faulty and/orfailing. The improved virtual model of the present invention embodied asan AVM may be updated, either periodically or on one off occasions, suchas during a service call or update call.

In some embodiments, a fourth dimension in addition to an XYZ dimensionwill include date and time and allow for an historical view of a life ofa structure to be presented in the virtual model. Accordingly, in someembodiments, onsite cameras and/or Sensors may be deployed and data maybe gathered from the on-site cameras and Sensors either periodically orupon command. Data gathered may be incorporated into the improvedvirtual model.

In still another aspect, the AVM may aggregate data across multipleproperties and buildings. The aggregated data may include conditionsexperienced by various buildings and mined or otherwise analyzed, suchas via artificial intelligence and unstructured queries. Accordingly,the AVM may quantify reasons relating to one or more of: how toreposition machines, route workflow or otherwise improve, designs thatwork well; designs that fail; popular aspects; generate multiple VirtualModels with various quantified features; original and modified modelversions and almost any combination thereof.

Although data may be gathered in various disparate and/or related ways,an aggregate of data may be quickly and readily accessed via thecreation of indexes. Accordingly, indexes may be according to one ormore of: date/time stamp; feature; popularity; cost; User specificquery; Plumbing; Electrical; HVAC; Structural aspects; Access areas;Periodic data and position capture with camera/Sensor attached to afixed position; during construction; during modification; duringDeployment; airflow; HVAC; machinery; traffic flows during use ofstructure; audible measurements for noise levels; and almost any otheraspect of captured data.

In another aspect, an Augmented Virtual Model may receive datadescriptive of generally static information, such as, one or more of:product specifications, building material specifications, productmanuals, and maintenance documentation.

Generally static information may be utilized within the AugmentedVirtual Model to calculate Performance of various aspects of a property.Dynamic data that is captured during one of: a) design data; b) builddata; and c) deployed data, may be used to analyze actual Performance ofa property and also used to update an Augmented Virtual Model andincrease the accuracy of additional predictions generated by theAugmented Virtual Model. Maintenance records and supportingdocumentation may also be archived and accessed via the AVM. A varietyof Sensors may monitor conditions associated with one or both of thestructure and the parcel. The Sensors and generated data may be used toextrapolate Performance expectations of various components included inthe Augmented Virtual Model. Sensor data may also be aggregated withSensor data from multiple Augmented Virtual Model models from multiplestructures and/or properties and analyzed in order to track and/orpredict Performance of a structure or model going forward.

GLOSSARY

“Agent” as used herein refers to a person or automation capable ofsupporting a Smart Device at a geospatial location relative to a GroundPlane.

“Ambient Data” as used herein refers to data and data streams capturedin an environment proximate to a Vantage Point and/or an equipment itemthat are not audio data or video data. Examples of Ambient Data include,but are not limited to Sensor perception of: temperature, humidity,particulate, chemical presence, gas presence, light, electromagneticradiation, electrical power, moisture and mineral presence.

“Analog Sensor” and “Digital Sensor” as used herein include a Sensoroperative to quantify a state in the physical world in an analogrepresentation.

“As Built” as used herein refers to details of a physical structureassociated with a specific location within the physical structure orparcel and empirical data captured in relation to the specific location.

“As Built Features” as used herein refers to a feature in a virtualmodel or AVM that is based at least in part upon empirical data capturedat or proximate to a correlating physical location of the feature.Examples of As Built Features include placement of structural componentssuch as a wall, doorway, window, plumbing, electrical utility, machineryand/or improvements to a parcel, such as a well, septic, electric orwater utility line, easement, berm, pond, wet land, retaining wall,driveway, right of way and the like.

“As Built Imagery” (Image Data) as used herein shall mean image datagenerated based upon a physical aspect.

“Augmented Virtual Model” (sometimes referred to herein as “AVM”): asused herein is a digital representation of a real property parcelincluding one or more three dimensional representations of physicalstructures suitable for use and As Built data captured descriptive ofthe real property parcel. An Augmented Virtual Model includes As BuiltFeatures of the structure and may include improvements and featurescontained within a Processing Facility.

“Property” as used herein shall mean one or more real estate parcelssuitable for a deployed Processing Facility that may be modeled in anAVM.

“Directional Indicator” as used herein shall mean a quantification of adirection generated via one or both of: analogue and digitalindications.

“Directional Image Data” as used herein refers to image data capturedfrom a Vantage Point with reference to a direction. Image data mayinclude video data.

“Directional Audio” as used herein refers to audio data captured from aVantage Point within or proximate to a property and from a direction.

“Deployment” as used herein shall mean the placement of one or more of:facility machinery and an equipment item into operation.

“Deployment Performance” as used herein shall mean one or both of:objective and subjective quantification of how one or more of: facility,machinery and an equipment item operated, which may be depicted in anAVM.

“Design Feature” as used herein, shall mean a value for a variabledescriptive of a specific portion of a property. A Design Feature mayinclude, for example, a size and shape of a structural element or otheraspect, such as a doorway, window or beam; a material to be used, anelectrical service, a plumbing aspect, a data service, placement ofelectrical and data outlets; a distance, a length, a number of steps; anincline; or other discernable value for a variable associated with astructure or property feature.

“Digital Sensor” as used herein includes a Sensor operative to quantifya state in the physical world in a digital representation.

“Experiential Data” as used herein shall mean data captured on orproximate to a subject Processing Facility descriptive of a conditionrealized by the Processing Facility. Experiential data is generated byone or more of: digital and/or analog sensors, transducers, imagecapture devices, microphones, accelerometers, compasses and the like.

“Experiential Sensor Reading” as used herein shall mean a value of asensor output generated within or proximate to a subject ProcessingFacility descriptive of a condition realized by the Processing Facility.An Experiential Sensor Reading may be generated by one or more of:digital and/or analog sensors, transducers, image capture devices,microphones, accelerometers, compasses and the like.

“Ground Plane” as used herein refers to horizontal plane from which adirection of interest may be projected.

“Image Capture Device” or “Scanner” as used herein refers to apparatusfor capturing digital or analog image data, an Image capture device maybe one or both of: a two dimensional camera (sometimes referred to as“2D”) or a three dimensional camera (sometimes referred to as “3D”). Insome examples an Image Capture Device includes a charged coupled device(“CCD”) camera. An Image Capture Device may also be capable of taking aseries of images in a short time interval and associating the imagestogether to create videos for use in four-dimensional model embodiments.

“Infrastructure” as used herein refers to a manmade or automation mademanufacture suitable for deployment to meet a basic service of asociety, and installations needed for the functioning of a community orsociety, such as transportation and communications systems, water andpower lines.

“Intelligent Automation” as used herein refers to a logical processingby a device, system, machine or equipment item (such as data gathering,analysis, artificial intelligence, and functional operation) andcommunication capabilities.

“Lag Benefit” as used herein shall mean a benefit derived from, or inrelation to a Lead Action.

“Lead Actions” as used herein shall mean an action performed on, in, orin relation to a structure to facilitate attainment of a PerformanceLevel.

“Performance” as used herein may include a metric of an action orquantity. Examples of Performance may include metrics of: number ofprocesses completed, energy efficiency; length of service; cost ofoperation; quantity of goods processed or manufacture; quality of goodsprocessed or manufacture; yield; and human resources required.

“Performance Level” as used herein shall mean one or both of a quantityof actions executed and a quality of actions.

“Processing Facility” as used herein shall mean a structure “QualityLevel” capable of receiving in a processing material and/or a consumableand outputting a product.

“Ray” as used herein refers to a straight line including a startingpoint and extending indefinitely in a direction.

“Sensor” as used herein refers to one or more of a solid state,electro-mechanical, and mechanical device capable of transducing aphysical condition or property into an analogue or digitalrepresentation and/or metric.

“Smart Device” as used herein includes an electronic device including,or in logical communication with, a processor and digital storage andcapable of executing logical commands.

“Structure” as used herein refers to a manmade assembly of partsconnected in an ordered way. Examples of a Structure in this disclosureinclude a building; a sub-assembly of a building; a bridge, a roadway, atrain track, a train trestle, an aqueduct; a tunnel a dam, and aretainer berm.

“Structural Message” as used herein refers to a logical communicationgenerated by automation (such as a sensor or machine) incorporated into,affixed to or operated within or proximate to a structure.

“Structural Messaging” as used herein refers to an action that generatesand/or transmits a Structural Message.

“Total Resources” as used herein shall mean an aggregate of one or moretypes of resources expended over a time period.

“Transceive” as used herein refers to an act of transmitting andreceiving data.

“Transceiver” as used herein refers to an electronic device capable ofone or both of transmitting and receiving data.

“Vantage Point” as used herein refers to a specified location which maybe an actual location within a physical facility or a virtualrepresentation of the actual location within a physical facility.

“Vector” as used herein refers to a magnitude and a direction as may berepresented and/or modeled by a directed line segment with a length thatrepresents the magnitude and an orientation in space that represents thedirection.

“Virtual Processing Facility” (“VPS”): as used herein shall mean adigital representation of a physical structure suitable for use. TheVirtual Processing Facility may include Design Features and As BuiltFeatures. The Virtual Processing Facility may be included as part of anAVM.

“Vital Condition” as used herein refers to a condition measurable via adevice or Sensor located in or proximate to a structure, wherein a valueof the measured condition is useful to determine the structure's abilityto meet a set of predetermined conditions.

“WiFi” as used herein shall mean a communications protocol with theindustrial, scientific, and medical (“ISM”) radio bands within thefrequency range of 6.7 MHz-250 GHz. This shall be interpreted to includeultra-wideband frequencies.

According to the present invention, multiple automated sensing devicesare deployed in or proximate to a structure to provide data quantifyingrespective conditions registered by the respective sensors. The dataquantifying respective conditions registered by the respective sensorsis referenced to generate a status and/or condition of one or more of: adeployed structure, a structure in the process of being built; and/or astructure in the process of being retrofitted.

The present invention includes an automated system to coordinate datageneration, data recording, data communication and overall control ofsensor operation and data collection from the sensors. One or both ofSensor Clusters and Sensor Gateways provide a platform for coordinationamongst multiple sensors and enables numerous methods useful to evaluatea status and/or condition of a related structure.

In some embodiments, a location of one or more sensors may be generatedaccording to the methods herein. The location may be in relation to oneor more of: a home position; a position of an Agent; and a position ofone or more Reference Position Transceivers. An Agent may be guided to asensor and/or an area of interest based upon a sensor reading usingorienteering methods and apparatus presented herein. For example, acontroller may receive sensor data quantifying temperature and humiditythat exceed an optimal range of temperature and humidity (e.g. the dataquantifying temperature and humidity may indicate an environmentconducive to termites in the Structure, or simply inefficient insulationfrom an outside environment). Using Orienteering, an Agent may be guidedto one or both of the sensor that generated the data and an area ofinterest indicated by the measured data. A user interface may includehuman ascertainable indications of the conditions quantified and/or thelocation of the conditions quantified.

Additional example may include guiding an Agent to a sensor to replace apower source, such as a battery or battery pack. Other exemplary powersources include an antenna or array of antennas tuned to receive ambientenergy and recharge an energy storage device (such as a battery).

Referring now to FIG. 1A a block diagram illustrates various aspects ofthe present invention and interactions between the respective aspects.The present invention includes an Augmented Virtual Model 111 of aProcessing Facility that includes As Built Features. The generation andinclusion of As Built Features, based upon location and directionspecific data capture, is discussed more fully below. Data may betransmitted and received via one or both of digital and analogcommunications, such as via a wireless communication medium 117.

According to the present invention, one or more Deployment PerformanceMetrics 112 are entered into automated apparatus in logicalcommunication with the AVM 111. The Deployment Performance Metric 112may essentially include a purpose to be achieved during Deployment of amodeled Processing Facility. By way of non-limiting example, aDeployment Performance Level may include one or more of: a production orquantity; quality; yield; scalability; a level of energy efficiency; alevel of water consumption; mean time between failure for equipmentincluded in the Processing Facility; mean time between failure formachinery installed in the structure; a threshold period of time betweenrepairs on the Processing Facility; a threshold period of time betweenupgrades of the Processing Facility; a target market value for aproperty; a target lease or rental value for a property; a cost offinancing for a property; Total Cost of ownership of a property; TotalCost of Deployment of a property or other quantifiable aspect.

In some embodiments, Deployment Performance Metrics may be related to afungible item, such as a measurement of energy (KWH of electricity,gallon of fuel oil, cubic foot of gas, etc.); man hours of work; trademedium (i.e. currency, bitcoin, stock, security, option etc.); parts ofmanufactures volume of material processed or other quantity. Relatingmultiple disparate Deployment Performance Metrics to a fungible itemallows disparate Performance Metrics to be compared for relative value.

Modeled Performance Levels 113 may also be entered into the automatedapparatus in logical communication with the AVM 111. The ModeledPerformance Levels 113 may include an appropriate level of Performanceof an aspect of the structure in the AVM affected by the DeploymentPerformance Metric 112. For example, a Performance Level 113 for energyefficiency for a structure modeled may include a threshold of KW hoursof electricity consumed by the structure on a monthly basis. Similarly,a target market value or lease value may be a threshold pecuniaryamount. In some embodiments, a pecuniary amount may be according to aperiod of time, such as monthly, or a term of years.

Empirical Metrics Data 114 may be generated and entered into theautomated apparatus on an ongoing basis. The Empirical Metrics Data 114will relate to one or more of the Deployment Performance Metrics and maybe used to determine compliance with a Deployment Performance Leveland/or a Performance Levels. Empirical Metrics Data 114 may include, byway of non-limiting example, one or more of: a unit of energy; an unitof water; a number of service calls; a cost of maintenance; a cost ofupgrades; equipment details, design details, machinery details,identification of human resources deployed; identification oforganizations deployed; number of human resources; demographics of humanresources (i.e. age, gender, occupations, employment status, economicstatus, requiring assistance with basic living necessities; and thelike); percentage of time structure is occupied; purpose of occupancy(i.e. primary residence, secondary residence, short term rental, longterm lease, etc.); Sensor readings (as discussed more fully below); manhours required for structure repair/maintenance/upgrades; total currency(or other fungible pecuniary amount) expended on behalf of a structureor property.

In addition to Empirical Metrics Data 114, Lead Actions and expected LagBenefits 115 that may cause an effect on one or both of a DeploymentPerformance Level 112 and a Performance Level 113, may be entered intothe automated apparatus. A Lead Action may include an action expected toraise, maintain or lower an Empirical Metrics Data 114. For example, anaction to install water efficient plumbing fixtures may be scheduled inorder to improve water consumption metrics. Similar actions may relateto electrically efficient devices, or automatic electric switches beinginstalled; preventive maintenance being performed; structure automationdevices being installed and the like. Other Lead Actions may includelimiting a demographic of occupants of a structure to a certaindemographic, such as senior citizens. An expected benefit may bemeasured in Lag Benefit measurements, such as those described asEmpirical Metrics Data 114, or less tangible benefits, such as occupantsatisfaction.

The automated apparatus may also be operative to calculate FuturePerformance 116 based upon one or more of: AVM Model with As Built Data111; Deployment Performance Metrics 112; Modeled Performance Levels 113and Empirical Metrics Data 114. Future Performance may be calculated interms of an appropriate unit of measure for the aspect for whichPerformance is calculated, such as, for example: an energy unit; manhours; mean time between failures and dollar or other currency amount.

Calculation of Future Performance 116 may be particularly useful tocalculate Total Resources calculated to be required to support aparticular structure, group of structures, properties and/or group ofproperties over a term of years (“Total Resources Calculated”). TotalResources Calculated may therefore be related to calculations of FuturePerformance 116 and include, for example, one or more of: energy units;water units; man hours; equipment; machinery and dollars (or othercurrency or fungible item). In some embodiments, calculations of FuturePerformance may include a Total Cost of Ownership for a term of years.For example, a Total Cost of Ownership for a property may include apurchase amount and amounts required for maintenance, repair andupgrades from day one of Deployment through twenty years of Deployment(a shorter or longer term of years may also be calculated).

Accordingly, some embodiments may include a calculation of TotalResources required that includes a purchase price of a property with aProcessing Facility, that incorporates a total cost associated with theproperty over a specified term of years. The total cost will be basedupon the AVM with As Built Data 111; Deployment Performance Metrics 112;Modeled Performance Levels 113 and Empirical Metrics Data 114.

Moreover, Total Resources required may be aggregated across multipleproperties and. Structures. Aggregation of properties may be organizedinto property pools to mitigate risk of anomalies in the Calculation ofFuture Performance. Of course, the benefits of property ownership and/ormanagement may also be pooled and compared to the Total Resourcesrequired. In various embodiments, different aspects of calculated FuturePerformance 116 may be aggregated and allocated to disparate parties.For example, first aggregation may relate to man hours of techniciantime for structure repair and maintenance and the fulfillment ofobligations related to the aggregation may be allocated to a firstparty. A second aggregation may relate to machinery Performance andobligations allocated to a second party. A third aggregation may relateto equipment Performance and obligations allocated to a third party.Other aggregations may similarly be allocated to various parties. Insome embodiments, financial obligations incorporating one or both ofacquisition cost and ongoing Deployment costs may be allocated andfinanced as a single loan. Other embodiments include a calculated FuturePerformance cost being incorporated into a purchase price.

An important aspect of the present invention includes definition andexecution of Lead Actions based upon one or more of: the AVM Model withAs Built Data 111; Deployment Performance Metrics 112; ModeledPerformance Levels 113; Empirical Metrics Data 114 and Calculations ofFuture Performance 116.

Referring now to FIG. 1B, an AVM is generally associated with a propertythat includes real estate parcels 140-143. According to someembodiments, one or more of: monitoring; service call; an improvement, arepair, maintenance and an upgrade are performed on the property. Theproperty is identified according to an automated determination of alocation and a particular position, elevation and direction are furtherdetermined automatically within the property. Smart Devices may be usedto access data records stored in an AVM according to a unique identifierof a physical location of the real estate parcels 140-143.

As illustrated, a map of real estate parcels 140-143 is shown with icons140A-142A indicating parcels 140-142 that have virtual structures140A-142A included in a virtual model associated with the parcels. Otherparcels 143 have an indicator 143A indicating that a virtual model is inprocess of completion.

In some methods utilized by the present invention, data in an AVM may beaccessed via increasingly more accurate determinations. A first level ofgeospatial location determinations may be based upon the real estateparcels 140-143 themselves and a second geospatial determination may bemade according to Reference Position Transceivers (discussed more fullybelow) included within the boundaries of the real estate parcels140-143. Still more accurate location position may be calculatedaccording to one or both of a direction determination and anaccelerometer or other location determination technology. Accordingly,it is within the scope of the present invention to access a record of adesign model for a specific wall portion within a structure based uponidentification of a particular parcel of real estate parcels 140-143 anda location within a structure situated within the real estate parcels140-143 and height and direction. Likewise, the present inventionprovides for accessing As Built data and the ability to submit As Builtdata for a specific portion of a structure based upon an accurateposition and direction determination.

For example, in some embodiments, a first level of locationidentification may include a property 141-143 identified based upon afirst wireless communication modality, such as a GPS communication. Asecond level of location identification may include a structure141A-143A identified via one or more of GPS; UWB; WiFi; soniccommunications; and Bluetooth communications. A third level of locationidentification may include an Agent position within a structure (orproperty) based upon logical communications via one or more of: UWB;WiFi; sonic communications; and Bluetooth communications. A fourth levelof location identification may include a determination of a distancefrom a surface proximate to an Agent, the distance based upon infraredand/or sonic transceiving.

In some implementations of the present invention, a property uniqueidentifier may be assigned by the AVM and adhere to a standard foruniversally unique identifiers (UUID), other unique identifiers may beadopted from, or be based upon, an acknowledged standard or value. Forexample, in some embodiments, a unique identifier may be based uponCartesian Coordinates, such as global positioning system (GPS)coordinates. Other embodiments may identify a property according to oneor both of: a street address and a tax map number assigned by a countygovernment of other authority.

In some embodiments, an AVM may also be associated with a larger groupof properties, such as a manufacturing plant, research and development,assembly, a complex, or other defined arrangement.

As illustrated, in some preferred embodiments, an electronic recordcorrelating with a specific property may be identified and then accessedbased upon coordinates generated by a GPS device, or other electroniclocation device. The GPS device may determine a location and correlatethe determined location with an AVM record listing model data, As Builtdata, improvement data, Performance data, maintenance data, cost ofoperation data, return on investment data and the like.

In another aspect data generated by sensors deployed in a structure maybe aggregated and analyzed according to a property location and/orstructure location associated with the Sensor/Sensor Cluster/SensorGateway. In this manner, an event may be tracked in a larger geographicarea with numerous data points. For example, an event such as the launchof a rocket may cause data to be generated by multiple Sensor/SensorCluster/Sensor Gateways and tracked across a geographic area. Similarly,a natural event, such as an earthquake, hurricane, wildfire and the likemay be tracked with highly accurate sensor data across tens, hundreds ormany thousands of data points. Still other events may include, forexample, power usage, power generation, water flow in a hydroelectricsystem, water management in a reservoir system, flooding, release oftoxic components into the environment etc.

Referring now to FIG. 1C, a relational view of an Augmented VirtualModel 100 with a Virtual Processing Facility 102B is illustrated, aswell as a physical structure 102A. The Augmented Virtual Model 100includes a virtual model stored in digital form with a design aspectthat allows for a physical structure 102A suitable for use to bedesigned and modeled in a virtual environment. The design aspect mayreference Performance data of features to be included in a VirtualProcessing Facility 102B and also reference variables quantifying anintended use of the Virtual Processing Facility 102B. The VirtualStructure 102B and the Augmented Virtual Model 100 may reside in avirtual setting via appropriate automated apparatus 108. The automatedapparatus 108 will typically include one or more computer servers andautomated processors as described more fully below and may be accessiblevia known networking protocols.

The Physical Structure 102A may include transceivers 120 or other typeof sensor or transmitter or receivers that monitor an area of ingressand egress 122, such as a doorway, elevator and/or loading dock.Reference point transceivers 121A may be used as wireless references ofa geospatial position. A wireless communication device 123 may also linklogical infrastructure within the structure 102A with a digitalcommunications network.

In correlation with the design aspect, the present invention includes anAs Built Model 101 that generates a Virtual Structure 102A in thecontext of the Augmented Virtual Model 100. The As Built Model 101includes virtual details based upon As Built data captured on orproximate to a physical site of a related physical structure 102A. TheAs Built data may be captured, for example, during construction ormodification of a physical structure 102A.

The As Built Model 101 may include detailed data including imagecaptures via one or more image capture devices 107 and physicalmeasurements of features included in the physical structure 102A. Thephysical measurements may be during a build phase of the physicalstructure; or subsequent to the build phase of the physical structure.In some embodiments, original As Built measurements may be supplementedwith additional data structure data associated with repairs orimprovements are made to the physical structure. Details of recordablebuild aspects are placed as digital data on a recordable medium 104included in the automated apparatus 108.

The digital data included on a recordable medium 104 may thereforeinclude, for example, one or more of: physical measurements capturingExperiential Data; image data (i.e. digital photos captured with a CCDdevice); laser scans; infra-red scans and other measurement mediums. Oneor more records on the recordable medium 104 of an As Built structuremay be incorporated into the Augmented Virtual Model 100 therebymaintaining the parallel nature of the Augmented Virtual Model 100 withthe physical structure 102A.

In some embodiments, As Built data on a recordable medium 104 may begenerated and/or captured via an image capture device 107.

As the physical structure is deployed for use, subsequent measurementsthat generate and/or capture Experiential Data may be made andincorporated into the Augmented Virtual Model 100. In addition, a usermay access and update 103 the Augmented Virtual Model 100 to ascertainfeatures of the physical structure 102A that have been virtuallyincorporated into the Augmented Virtual Model 100. In some examples, atablet, handheld network access device (such as, for example a mobilephone) or other device with automated location service may be used todetermine a general location of a physical structure 102A. For example,a smart phone with global positioning system (GPS) capabilities may beused to determine a physical address of a physical structure, such as123 Main Street. Stored records containing data relating to 123 MainStreet may be accessed via the Internet or other distributed network.

In addition to the use of GPS to determine a location of a User Device,the present invention provides for a real estate parcel with a physicalstructure 102A that includes more radio frequency (or other mechanism)location identifiers 121A. Location identifiers 121A may include, forexample, radio transmitters at a defined location that may be used toaccurately identify via triangulation, a position of a user device 106,such as a: tablet, smart phone or virtual reality device. The positionmay be determined via triangulation, single strength, time delaydetermination or other process. In some embodiments, triangulation maydetermine a location of a user device within millimeters of accuracy.

Other location identifiers may include, by way of non-limiting example,RFID chips, a visual markings (i.e. a hash tags or barcode), pins orother accurately placed indicators. Placement of the locationidentifiers may be included in the AVM and referenced as the location ofthe physical user device is determined. As described above, specificlocation identifiers may be referenced in the context of GPS coordinatesor other more general location identifiers.

Based upon the calculated location of the user device 106, details ofthe physical structure 102A may be incorporated into the VirtualStructure 102B and presented to a user via a graphical user interface(GUI) on the user device 106.

For example, a user may approach a physical structure and activate anapp on a mobile user device 106. The app may cause the user device 106to activate a GPS circuit included in the user device and determine ageneral location of the user device 106, such as a street addressdesignation. The general location will allow a correct AVM 100 to beaccessed via a distributed network, such as the Internet. Once accessed,the app may additionally search for one or more location identifiers121A of a type and in a location recorded in the AVM. An AVM mayindicate that one or more RFID chips are accessible in a kitchen, aliving room and each bedroom of a structure. The user may activateappropriate Sensors to read the RFID chips and determine their location.In another aspect, an Augmented Virtual Model 100 may indicate thatlocation identifiers 121A are placed at two or more corners (or otherplacement) of a physical structure 102A and each of the locationidentifiers 121A may include a transmitter with a defined location andat a defined height. The user device 106, or other type of controller,may then triangulate with the location identifiers 121A to calculate aprecise location and height within the physical structure.

Similarly, a direction may be calculated via a prescribed movement ofthe user device 106 during execution of code that will record a changein position relative to the location identifiers 121A. For example, auser smart device, such as a smart phone or user device 106 may bedirected towards a wall or other structure portion and upon execution ofexecutable code, the smart device may be moved in a generally tangentialdirection towards the wall. The change in direction of the user device106 relative to the location identifiers 121A may be used to calculate adirection. Based upon a recorded position within the structure 102A andthe calculated direction, a data record may be accessed in the AugmentedVirtual Model 100 and a specific portion of the Augmented Virtual Model100 and/or the Virtual Structure 102B may be presented on the userdevice 106. In other embodiments, a direction may be made, or verifiedvia a mechanism internal to the smart device, such as a compass oraccelerometer.

In still another aspect of the present invention, in some embodiments,transmissions from one or more location identifiers 121A may becontrolled via one or more of: encryption; encoding; passwordprotection; private/public key synchronization or other signal accessrestriction. Control of access to location identifiers 121A may beuseful in multiple respects, for example, a location identifier mayadditionally function to provide access to data, a distributed networkand/or the Internet.

The Virtual Structure 102B may include one or both of: historical dataand most current data relating to aspects viewable or proximate to theuser device 106 while the user device is at the calculated location inthe physical structure 102A. In this way, the parallel virtual world ofthe Augmented Virtual Model 100 and the Virtual Structure 102B maypresent data from the virtual world that emulates aspects in thephysical world, and may be useful to the user accessing the user device106, while the user device is at a particular physical location. Asdiscussed within this document, data presented via the Augmented VirtualModel 100 may include one or more of: design data, As Built data,Experiential Data, Performance data relating to machinery and/orfeatures of the Augmented Virtual Model 100 or physical structure;maintenance data, and annotations.

Annotations may include, for example, a user's or designer's noterecorded at a previous time, a service bulletin, maintenance log,operation instructions or a personal note to a subsequent user, such asa virtual “John Smith was here” such guest log indicating who hadfrequented the location. Annotations may include one or both of text andimage data. For example, an annotation may include an image of thelocation captured at a given time and date. The image may be of apersonal nature, i.e. the living room while the Smith's owned thestructure, or a professional nature, i.e. the living room after beingpainted by XYZ Contractor on a recorded date. In some embodiments,annotations may be used to indicate completion of a work order.Recordation of completion of a work order may in turn trigger a paymentmechanism for paying an entity contracted to complete the work order. Inanother aspect, annotations may relate to an AVM or a Virtual Structureas a whole, or to a particular aspect that is proximate to a location ofthe user device within the Virtual Structure.

In some embodiments, details of a proposed use of a structure and parcelmay be input into a design module and used to specify or recommendfeatures to be included in an Augmented Virtual Model 100.

According to the present invention, features of a Structure and parcelare generated within a digital design model and then tracked as thefeatures are implemented in a build process and further tracked inPerformance of the structure as it is placed into use. To the extentavailable, Performance is tracked in the context of variables relatingto use. Variables may include, for example: a use of the structure, suchas manufacturing and/or processing; a number of resources accessing in astructure; demographics of the human resources; number of months peryear the structure is deployed for use; which months of the year astructure is deployed for use; which hours of the day the structure isoccupied and other relevant information.

As Experiential Sensor Readings are generated, they may be memorializedto generate Experiential Data associated with a physical structure 102A.The Experiential Data is collected and analyzed via structured queriesand may also be analyzed with Artificial Intelligence processes such asunstructured queries to derive value. In some embodiments, ExperientialData may also be associated with a human and/or an animal interactingwith the structure 102A. Whereas former process plants were generallydesigned and built to mitigate against variability in a human 118 andbetween disparate humans 118. The present invention allows for humanvariability to be monitored via sensors within device 119 and thestructure to be modified to optimally inter-relate with the values forvariables attributable to a human 118 that will inhabit or otherwiseinteract with the structure 102A. Human (and/or animal) may bequantified with sensors within device 119 installed on or proximate tothe Human 118. Alternatively, sensors 124 located in, or proximate to, astructure 102A may be used to monitor human variability. Biosensors maybe used to provide empirical data of humans 118 interacting with astructure may be analyzed using structured or unstructured queries todevice relationships between structure performance and human biometrics.Accordingly, sensors may be used to quantify interaction between a human118 and an As Built structure 102A according to physiological andbehavioral data, social interactions, and environmental factors withinthe structure, actions undertaken, movements, and almost anyquantifiable aspect.

As Built Features and biometrics may be further utilized to controlvarious structure automation devices. Structure automation devices mayinclude, by way of non-limiting example one or more of: automated locksor other security devices; thermostats, lighting, heating, chemicalprocessing, cutting, molding, laser shaping, 3D printing, assembly,cleaning, packaging and the like. Accordingly, a structure with recordedAs Built design features and vibration sensors may track activities in astructure and determine that a first occupant associated with a firstvibration pattern of walking is in the structure. Recorded vibrationpatterns may indicate that person one is walking down a hallway andautomatically turn on appropriated lighting and adjust one or more of:temperature, sound and security. Security may include locking doors forwhich person one is not programmed to access. For example, a firstpattern of vibration may be used to automatically ascertain that aperson is traversing an area of a structure for which a high level ofsecurity is required or an area that is designated for limited accessdue to safety concerns. As Built data has been collected. Otherstructure automation may be similarly deployed according to As Builtdata, occupant profiles, biometric data, time of day, or othercombination of available sensor readings.

Referring now to FIG. 1D, according to the present invention a virtualmodel 100 is generated that correlates with a physical facility 102A andincludes virtual representations of As Built features and ExperientialData. As discussed more fully herein, the virtual model may include anAVM with As Built data, such as image data and measurements, includedwithin the model. In addition, sensor data may be collected over timeand incorporated into the AVM. The AVM may include virtualrepresentations of one or more of: sensors 155; equipment 156-158;controls 161; infrastructure 159, such as HVAC, utilities, such aselectric and water, gas lines, data lines, etc. and vantage points 151.

In some implementations, a virtual reality headset may be worn by a userto provide an immersive experience from a vantage point 151 such thatthe user will experience a virtual representation of what it would belike to be located at the vantage point 151 within the facility 152 at aspecified point in time. The virtual representation may include acombination of Design Features, As Built Data and Experiential Data. Avirtual representation may therefore include a virtual representation ofimage data via the visual light spectrum, image data via infrared lightspectrum, noise and vibration reenactment. Although some specific typesof exemplary sensor data have been described, the descriptions are notmeant to be limiting unless specifically claimed as a limitation and itis within the scope of this invention to include a virtualrepresentation based upon other types of captured sensor data may alsobe included in the AVM virtual reality representation.

Referring now to FIG. 1E, a user 131 is illustrated situated within anAVM 111. The user 131 will be virtually located at a Vantage Point 137and may receive data 136, including, but not limited to one or more of:image data 134, audio data 135 and Ambient Data 136. The user 131 mayalso be provided with controls 133. Controls 133 may include, forexample, zoom, volume, scroll of data fields and selection of datafields. Controls may be operated based upon an item of Equipment 132within a Field of View 138 of the User 131 located at a vantage point137 and viewing a selected direction (Z axis). The user is presentedwith Image Data from within the AVM 111 that includes As Built data andvirtual design data.

Additional examples may include sensor arrays, audio capture arrays andcamera arrays with multiple data collection angles that may be complete360 degree camera arrays or directional arrays, for example, in someexamples, a sensor array (including image capture sensors) may includeat least 120 degrees of data capture, additional examples include asensor array with at least 180 degrees of image capture; and still otherexamples include a sensor array with at least 270 degrees of imagecapture. In various examples, data capture may include sensors arrangedto capture image data in directions that are planar or oblique inrelation to one another.

Referring now to FIG. 2, a functional block illustrates variouscomponents of some implementations of the present invention. Accordingto the present invention automated apparatus included in the AVM 201 areused to generate a model of a Virtual Structure (“VPS”) and may alsoincorporate a model and associated real estate parcel (“VPS”). One ormore pieces of equipment that will be deployed in the property may beincluded into the augmented virtual model 201, equipment may include,for example: machinery 211; building support items 212, and utilitiessupport 213. The AVM 201 may model operational levels 204 duringdeployment of a facility and associated machinery and equipment includedin the AVM 201. Machinery 211 may include, for example, manufacturingtools, robots or other automation, transport tools, chemical processingmachine, physical processing machine, assembly machine, heat processingmachine, cooling machine, deposition device, etching device, weldingapparatus, cutting apparatus, forming tool, drilling tool, shaping tool,transport machine, structure automation, air purification or filtersystems, noise containment device and the like. Utility supportequipment may include cabling, dish antennas, Wi-Fi, water softener,water filter, power, chemical supply, gas supply, compressed air supplyand the like, as well as uptime and downtime associated with a facilityutility and uptime and down time 243 of one or more aspects of thefacility.

The AVM 201 calculates a predicted Performance of the AVM and generatesOperational Levels 204 based upon the Performance 222, wherein“Performance” may include one or more of: total cost of deployment 214;operational experience 203 which may include one or both of: objectiveempirical measurements and satisfaction of operator's use an As Builtphysical model based upon the AVM; operational expectations 204, totalmaintenance cost 206, and residual value of an As Built following a termof years of occupation and use of an As Built Facility based upon theAVM. Performance 221 may also be associated with a specific item ofmachinery 211.

In another aspect, actual Operational Experience 203 may be monitored,quantified and recorded by the AVM 201. Data quantifying the OperationalExperience 203 may be collected, by way of non-limiting example, fromone or more of: Sensors incorporated into an As Built structure;maintenance records; utility records indicating an amount of energy 202(electricity, gas, heating oil) consumed; water usage; periodicmeasurements of an As Built structure, such as an infra-red scan ofclimate containment, air flow through air handlers, water flow, waterquality and the like; user surveys and maintenance and replacementrecords.

In still another aspect, a warranty 205 covering one or both of partsand labor associated with an As Built structure may be tracked,including replacement materials 207. The warranty 205 may apply to anactual structure, or one or more of machinery 211; building support 212item; and utility support item 213.

The AVM 201 may take into account a proposed usage of a Deployment of aStructure based upon values for Deployment variables and specify aspectsof one or more of: Machine's 211; building support 212; and utilitysupport 213 based upon one or both of a proposed usage and values forDeployment variables. Proposed usage may include, for example, how manyhuman resources will occupy a Structure, demographics of the resourcesthat will occupy the Structure; percentage of time that the Structurewill be occupied, whether the Structure is a primary residence, whetherthe Structure is a leased property and typical duration of leasesentered into, environmental conditions experienced by the Structure,such as exposure to ocean salt, Winter conditions, desert conditions,high winds, heavy rain, high humidity, or other weather conditions.

In another aspect, Deployment may relate to biometrics or other dataassociated with specific occupants of a structure. Accordingly, in someembodiments, sensors may monitor biologically related variables ofoccupants and/or proposed occupants. The biometric measurements may beused to determine one or both of Lead Actions and Lag Metrics. Leadactions may include one or more of: use of specific building materials,selection of design aspects; Deployment of structure equipment;Deployment of machinery; terms of a lease; length of a lease: terms of amaintenance contract; and structure automation controls.

According to the present invention, design aspects and structurematerials 210 may also be based upon the proposed usage and values forDeployment variables. For example, a thicker exterior wall with higherinsulation value may be based upon a structure's location in an adverseenvironment. Accordingly, various demographic considerations andproposed usage of a structure may be used as input in specifying almostany aspect of a Structure.

Total Cost of Deployment (TCD)

In still another consideration, a monetary value for one or more of: aTotal Cost of Deployment (“TCD”). Total maintenance cost (“TMC”) and adesired return on investment (“ROI”) for a property may be used as inputfor one or more design aspects included in an Augmented Virtual ModelSystem 200. Total Cost of Ownership, Total Maintenance Cost and ROI maybe used to determine optimal values of variables 202-205, 210-213specified in an Augmented Virtual Model System 200 and incorporated intoan As Built structure, and other improvements to a real estate parcel.

A Total Cost of Deployment 214 may change based upon a time period 215used to assess the Total Cost of Deployment 214. A ROI may include oneor more of: a rental value that may produce a revenue stream, a resalevalue, a cost of operation, real estate taxes based upon structurespecifications and almost any other factor that relates to one or bothof a cost and value.

Desirable efficiency and Performance may be calculated according to oneor more of: established metrics, measurement protocols and pastexperience. The AVM 201 and associated technology and software may beused to support a determination of a TCD. In another aspect, a TCD maybe based upon an assembly of multiple individual metrics, procedures toassess metrics, procedures to adjust and optimize metrics and proceduresto apply best results from benchmark operations. In the course ofmanaging Total Cost of Ownership, in some examples, initial steps mayinclude design aspects that model an optimal design based upon TotalCost of Ownership metrics and also model designed algorithms used toassess Total Cost of Ownership metrics.

In the following examples, various aspects of Total Cost of Deployment214, Total Maintenance Costs, and associated metrics, are considered inthe context of calculating a target Total Cost of Deployment 214.Accordingly, the AVM may be used to TCD optimization.

A designed Structure is ultimately built at a site on a real estateparcel. A build process may be specified and provide metrics that may beused in a process designed by an AVM 201 and also used as a physicalbuild proceeds. In some examples, time factors associated with aphysical build may be important, and in some examples time factorsassociated with a physical build may be estimated, measured and actedupon as they are generated in a physical build process. Examples of timefactors may include, one or more of: a time to develop and approve siteplans; a time to prepare the site and locate community providedutilities or site provided utilities; a time to lay foundations; a timeto build structure; a time to finish structure; a time to installinternal utilities and facilities related aspects; a time to install,debug, qualify and release equipment; times to start production runs andto certify compliance of production are all examples of times that canbe measured by various techniques and sensing equipment on a Structure'ssite. Various time factors for a build are valuable and may becomeincreasingly valuable as a physical build proceeds since the monetaryinvestment in the project builds before revenue flows and monetaryinvestments have clearly defined cost of capital aspects that scale withthe time value of money.

Various build steps may include material flows of various types.Material flow aspects may be tracked and controlled for cost andefficiency. Various materials may lower a build materials cost but raisetime factors to complete the build. Logical variations may be calculatedand assessed in an AVM 201 and optimal build steps may be generatedand/or selected based upon a significance placed upon various benefitsand consequences of a given variable value. Physical build measurementsand/or sensing on physical build projects may also be used as input inan assessment of economic trade-offs.

The equipment deployed may incur a majority of a build cost dependingupon user defined target values. The AVM may model and presentalternatives including one or more of: cost versus efficiency, quality240, time to build, life expectancy, market valuation over time. A costto build may be correlated with cost to deploy and eventual resale. Anoverall model of a Total Cost of Deployment 214 may include any or allsuch aspects and may also include external. In some examples, the natureof equipment trade-offs may be static, and estimations may be made fromprevious results. In some other examples, changes in technology,strategic changes in sourcing, times of acquisition and the like mayplay into models of Total Cost of Deployment 214.

In some examples, an initial efficiency of design which incurs largecosts at early stages of a project may have a dominant impact on TotalCost of Deployment 214 when time factors are weighted to real costs. Inother examples, the ability of a Structure to be flexible over time andto be changed in such flexible manners, where such changes areefficiently designed may dominate even if the initial cost aspects maybe less efficient due to the need to design in flexibility. As aStructure is built, and as it is operated the nature of changingcustomer needs may create dynamic aspects to estimations of Total Costof Deployment 214. Therefore, in some examples, estimates on theexpected dynamic nature of demands on a Structure may be modeled againstthe cost aspects of flexibility to model expectations of Total Cost ofDeployment 214 given a level of change.

In some examples, factors that may be less dependent on extrinsicfactors, such as product demand and the like may still be importantmetrics in Total Cost of Deployment 214. Included in the As Builtfactors may be calculations such as HVAC temperature load, in whichpersonnel and seasonal weather implications may be important. AVM modelsmay include a user interface to receive value useful in the AVM models.In addition, electronic monitoring, via Sensors that may determineenergy consumption, includes for example: electricity, fuel oil, naturalgas, propane and the like may be useful for estimation and measurement.

Temperatures may be monitored by thermocouples, semiconductor junctionbased devices or other such direct measurement techniques. In otherexamples, temperature and heat flows may be estimated based on photonbased measurement, such as surveying the Structure with infra-redimaging or the like.

Utility load may be monitored on a Structure wide basis and/or at pointof use monitoring equipment located at hubs or individual pieces ofequipment itself. Flow meters may be inline, or external to pipes wiresor conduits. Gases and liquid flows may be measured with physical flowmeasurements or sound based measurement. In other examples, electricitymay be monitored as direct current measurements or inferred inductivecurrent measurement.

In some examples, the nature and design of standard usage patterns of aStructure and an associated environment may have relevance to Total Costof Ownership. For example, usage that includes a larger number ofingress and egress will expose an HVAC system to increased load andusage that includes a significant number of waking hours withinhabitants in the building may incur increased usage of one or more of:machinery 211; building support devices 212; and utilities 234.

The nature and measurement aspects of vibration in the Structure mayalso be modeled and designed as the Structure is built. There may benumerous means to measure vibrations from capacitive and resistive basedmeasurements to optical based measurements that measure a subtle changein distance scale as a means of detecting vibration. Vibration mayresult from a Structure being located proximate to a roadway, train,subway, airport, tidal flow or other significant source of relativelyconsistent vibration. Vibration may also be more periodic, such asearthquake activity. In still another aspect, vibration may result fromhuman traffic within the property. The use of vibration monitoringSensors may indicate various activities that take place within thestructure and facilitate more accurate modeling of a life expectancy ofvarious aspects of the structure as well as machines located within thestructure.

Noise levels are another type of vibrational measurement which isfocused on transmission through the atmosphere of the Structure. In somecases, noise may emanate from one location after moving through solidstructure from its true source at another location. Thus, measurement ofambient sound with directional microphones or other microphonic sensingtypes may be used to elucidate the nature and location of noiseemanations. In some cases, other study of the noise emanations may leadto establishment of vibrational measurement of different sources ofnoise. Floors, ceilings, doorways, countertops, windows and otheraspects of a Structure may be monitored in order to quantify andextrapolate noise levels. Noise and vibrational measurement devices maybe global and monitor a region of a Structure, or they may be inherentlyincorporated into or upon individual equipment of the Structure.

In some examples, models of a Structure (including original models andAs Built models) may include routings of pipes, wires, conduits andother features of a Structure and the installed equipment that havestructure. Together with models of the building structure and theequipment placed in the building the various routed structures may bemarried in a detailed AVM 201.

In another aspect, an AVM 201 may include conflicts between the physicalstructures may be detected and avoided in the design stage at farimproved cost aspects. In some examples, a designer may virtuallyascertain a nature of the conflict and alter a design in virtual spaceto optimize operational aspects. Additionally, in some embodiments, anAs Built model may be generated during and after a Structure is builtfor various purposes. In some examples, a technician may inspect aStructure for conformance of the build to the designed model. In otherexamples, as an As Built Structure is altered to deal with neededchanges, changes will be captured and included in the As Built AVM 201.

In another aspect of the present invention, the AVM 201 may be used togenerate a virtual reality model of a property, including one or morestructures that may be displayed via user interface that includes animmersion of the user into a virtual setting. Immersion may beaccomplished, for example, via use of a virtual reality headset withvisual input other than a display screen is limited. In someembodiments, a virtual setting may be generated based upon a location ofthe user. For example, GPS coordinates may indicate a property and auser may wear a headset that immerses the user in a virtual realitysetting. The virtual reality setting may display one or more virtualmodels of structures that may be potentially constructed on theproperty.

Embodiments may include models generated, standard modeling softwaresuch as BIM 360™ field which may support the display of a Structuredesign in a very complete level of detail. Modeling of a Structure inits location or proposed location, or in multiple proposed locations,may be useful from a Total Cost of Ownership perspective, especiallyfrom an evaluation of the nature of a site layout including real estateproperty parcel options and the like.

In some examples, a virtual display observed in the field at the site ofan As Built or proposed build may allow for design changes and designevaluations to be viewed in a space before build is completed. Forexample, a structure may be completed to the extent that walls, floorsand ceilings are in place. A user may utilize a virtual display tounderstand the layout difference for different designs and the designsmay be iterated from designs with the least flexibility to more flexibleyet more complex designs.

In some examples, the design systems may include various types offeatures such as building structure, walls, ducts, utilities, pipes,lighting, and electrical equipment. The design systems are augmentedwith As Built Data and Experiential Data.

The design and modeling systems may be utilized to simulate and projectcost spending profiles and budgeting aspects. The modeling systems maytherefore be useful during the course of an audit, particularly whencomparing actual versus projected spending profiles. The comparison ofvarious spend sequencing may be used to optimize financing costs,maintenance, refurbishing and sequencing. The AVM 201 may be useful toprovide early estimates, and for cost tracking versus projections whichmay be visualized as displays across a virtual display of the building,facilities and equipment.

Energy/Utilities Cost: There may be numerous examples of tradeoffs insources of electric energy to a Structure. For example, a site may bedesigned with various utility supplies for power, with tailored powermanagement systems to balance the capacitance and impedance of theeffective load to minimize electricity cost. In addition, variousalternative forms of electric energy may be assessed and designed.Solar, geothermal and Wind generated electric power may make economicsense under certain conditions and may have time of day and seasonalrelevance. The design of flexible support facilities for theinstallation of initial energy generation capacity with provision forthe addition of additional capacity may be assessed. In some instances,backup power generation may be designed to ensure that a Structure mayrun at some level for a certain period of time. In some cases, this mayallow for continued production, in other examples, backup power may givea Structure the time to idle and shut down capacity in a safer and lessdamaging manner.

In some examples, an energy source for heating, cooling, humidificationand dehumidification equipment may be modeled and managed. In someexamples, a source of energy used may be one or more of electric,natural gas, propane, fuel oil or natural gas. Emergency backup may alsobe modeled and managed. Various choices between electric sources. Solarand fuel based energy consumption may be modeled and controlled based onupon market forecasts. Estimates may be periodically adjusted accordingto world and/or market events.

Enhanced inspection, and guidance capabilities enabled via ongoingelectronic Sensor measurements may facilitate one or more of:maintenance, expansion and optimization of Structure features, operationproperty equipment and maintenance models. Ongoing monitoring via Sensordata collection also increases knowledge of machines and operations, orother useful capacities towards knowing the state of the Structure.Decisions related to maintenance of equipment and facilities may beimportant decisions that modeling and operational management systemssupport. The various cost elements that may go into modeling mayinclude, for example, one or more variables related to consumables, suchas: a cost of consumables; frequency of replacement 241, quantity ofconsumables 242, life of replaced parts, nature of failures of differentpart types; manpower associated with planned and unplanned maintenanceand expected and actual life of equipment

Inside of a functional Structure, augmented reality functions viewablein an AVM 201 including an AVM may be used to guide operators,surveyors, repair workers, or other individuals, through the Structure.As one non-limiting example, a tablet, mobile device, or other smalldevice with a screen, imaging, and other sensing capabilities may beused in an augmented reality fashion towards this function.

As described above, facing a mobile device towards an area in aStructure and movement of the mobile device in a particular pattern maybe used to ascertain a specific area of the Structure for which AVM 201data should be accessed. A combination of one or more of: image,location, orientation, and other Sensors may also be used to identify tothe mobile device, which wall segment, building aspect, machinery orequipment the device is identifying. A location of mobile device, aheight and an angle of view may also be utilized to determine aspects ofthe structure for which a virtual model is being requested.

In some embodiments, a user may be presented with various layers ofdata, including, for example, one or more of: structural aspects of theStructure, plumbing, electrical, data runs, material specifications orother documentation, including but not limited to: basic identifyinginformation, installation information, service records, safety manuals,process records, expected service schedule, among many otherpossibilities.

A plurality of information may be thus easily accessible inside theStructure, and may be used for a variety of functions, including findinga specific machine to then diagnose and service a problem, regularinspection of equipment, guided tours of the Structure, or many otherfunctions. This information may be conveyed to the individual in aplurality of possible formats, such as lists that show up on the screen,clickable icons that show up next to the equipment in a Virtual Reality(“VR”) camera feed, or many other possibilities. These functions mayalso be accessible in a hands-free VR format with a VR headset, or othersuch device.

As the user is inside a Structure, the user may receive a plurality ofinformation, instructions, etc. while the user is proximate to thevarious aspects of the structures. For example, the user machinesthemselves, seeing them work, hearing the sounds they make, etc. tobetter inspect or service, among other possible functions, theStructure's equipment. With VR systems, similar travel, guidance, orinspection capabilities for a functional Structure may be achievedcompletely remotely from the Structure itself. Additionally, with VRsystems, these capabilities may occur prior, during, or after theconstruction and deployment of a Structure.

A VR system may constitute a headset or lens system with stereoscopicviewing capabilities, a sound conveying means, such as headphones, andvarious forms of user input, such as a handheld controller or footpedals as non-limiting examples. Various forms of imaging, surveying, ormodeling technology may be used to generate virtual models of afunctional Structure. As a non-limiting example, exploring such a modelwith a VR system may be used to examine layout, functioning, or otherparameters of a Structure before its construction. As an alternativenon-limiting example, exploring a model possibly generated by sensingtechnology in real time, or over a period of time prior to viewing witha VR system, may allow for inspection or demonstration capabilities in alocation entirely remotely from the actual Structure itself. This mayinclude both imagery and sounds captured within the Structure.

Collection of data may additionally include actual service lifeexperienced and performance of equipment used in an AVM which therebyenables enhanced modeling of a life expectancy of equipment included inan Augmented Virtual Model 100 and an As Built structure. VariousSensors may gather relevant data related to one or more of: use ofmachinery and equipment, performance of machinery items of equipment andan ambient environment inside or proximate to machinery and equipment.In addition, an unstructured query relating to the functioning or lifeexpectancy of equipment may be generated by a processor to access andinterpret data, thereby deriving relevant input to a decision makerbased upon analysis of the data.

Various examples of data to be acquired, relating to life expectancy ofequipment, may include, but is not limited to, hours of operation,conditions of operation (whether and how long the equipment may berunning under capacity, at rated capacity, or over capacity), or manyenvironmental conditions for operation; environmental conditions mayinclude the ambient temperature (or the difference in ambienttemperature from an ideal or other measured value), ambient humidity (orthe difference in ambient humidity from an ideal or other measuredvalue), ambient air particulate content (or a comparison of the currentair particulate level to a filter change schedule), presence orconcentration of ambient gasses (if relevant) such as carbon dioxide, orother gas, a number of times of ingress or egress into the Structurewhich may change ambient conditions or other trackable data.

Identification of Equipment

Identification capabilities may be facilitated or improved for one ormore of: structural aspects, machinery, equipment and utility supportwithin the Structure. This identification may take many forms throughvarious means of query and communication and may be facilitated throughvarious hardware and/or software means.

Non-limiting examples may include image based identification; a devicewith some imaging means, including but not limited to a mobile devicecamera, tablet device camera, computer camera, security camera, or ARheadset camera may image the equipment to be identified. Imagerecognition software may be used to identify the visualized equipment byits identifying features. Machine learning may be used to train systemsusing this software to identify specific features of the equipment inquestion. Other types of visual identifiers including but not limited toQR codes, may be used to visually identify equipment.

An additional non-limiting example may include location basedidentification; a device with some location means, including but notlimited to GPS, internal dead-reckoning, or other means, may be used todetermine a location within a Structure. Identifying information forequipment at or near the measured location may be accessed forassessment, based on its proximity to the location based signal.

An additional non-limiting example may also include direction basedidentification; with a fixed location, or in tandem with a locationmeans, a device may have capabilities to deduce orientation basedinformation of the device. This orientation information may be used todeduce a direction that the device is pointing in. This direction basedinformation may be used to indicate that the device is pointing to aspecific piece of equipment that may be identified.

An additional non-limiting example may also include As Built sensor andsensor generated experiential data based identification; identifyinginformation for various equipment may be stored and accessed within adatabase storing this information. This information may be accessed byvarious means by a user with certain qualification to that information.

An additional non-limiting example may include tag-based identification;identifying information for various equipment may be accessed throughproximity to many non-limiting examples of tagging capabilities, such asmagnetic tags, bar code tags, or others. These tags may contain theinformation in question or may reference the location of pertinentinformation to the owner, in order to convey this information to theowner.

An additional non-limiting example, data aggregation may include sensorsgenerating data that is associated with an IoT (Internet of Things)based identification. Various IoT devices (or Sensors) may include adigital storage, processor and transmitter for storing and conveyingidentifying information. Upon request, an IoT device may relayidentifying information of itself to a human with a communicatingdevice, or to its neighbors. It may also possibly convey informationreceived from and/or sent to other internet connected devices as well.

Data aggregated and stored for reference in calculation of Cost ofUpkeep considered in a TOC and may include data related to some or allof:

-   -   Documented items covered;    -   Long term warranty for Structure/building ownership;    -   Items included in purchase price;    -   financed amounts;    -   Tax implications;    -   Capital value;    -   Ability to expand Structure and/or structural features such as        baths or kitchens;    -   Lateral dimensions;    -   Vertical dimensions;    -   Building support systems;    -   Utilities;    -   Electric;    -   Water;    -   Discharge;    -   Aggregate Data;    -   Same Structure;    -   Multiple similar facilities;    -   Disparate Structure types;    -   Same geographic area;    -   Disparate geographic areas;    -   Locating Machine s and Equipment;    -   GPS (may be used in combination with other location        technologies;    -   Near field communication with reference point emitter in        Structure;    -   Wi-Fi;    -   RFID;    -   Reflector tags;    -   “Visual” recognition identifiers, i.e. hash, barcode; and    -   Directional—accelerometers in combination with visual        recognition identifiers.

As per the above listing, functionality may therefore include modeledand tracked Performance of a Structure and equipment contained withinthe Structure, including consumables 233 used and timing of receipt andprocessing of consumables; modeled and actual maintenance 232, includingquality of maintenance performed; equipment Performance includingyields; Consumables 233 tracking may include a frequency of replacementand quantity of replaced consumables; Utilities 234 tracking may includeprojected and actually units of energy consumed.

3D Scanning & Model Development

In one aspect of the present invention data related to the position andidentity of substantial elements of a Structure are first designed andthen recorded in their actual placement and installation. This mayinclude locations of building features, such as beams, walls, electricaljunctions, plumbing and etc. as the structure is designed andconstructed. As part of the Structure model, laser scanning may beperformed on site at various disparate times during construction. Aninitial scan may provide general information relating to the location ofthe structure in relationship to elements on the property such asroadways, utilizes such as electricity, water, gas and sewer to identifynon-limiting examples.

Additional events for scanning may occur during the construction processin order to capture accurate, three-dimensional (3D) “as-built” pointcloud information. Point cloud may include an array of points determinedfrom image capture and/or laser scanning or other data collectiontechnique of As Built features. In some examples, captured data may beconverted into a 3D model, and saved within a cloud-based data platform.

In some examples other methods of capturing spatially accurateinformation may include the use of drones and optical scanningtechniques which may include high resolution imagery obtained frommultiple viewpoints. Scanning may be performed with light based methodssuch as a CCD camera. Other methods may include infrared, ultraviolet,acoustic, and magnetic and electric field mapping techniques may beutilized.

Structure related information may include physical features generallyassociated with an exterior of a structure such as geo-location,elevation, surrounding trees and large landscaping features, undergroundutility locations (such as power, water, sewer, sprinkler system, andmany other possible underground utility features), paving, and pool orpatio areas. Structure related information may also include featuresgenerally related to a structure such as underground plumbing locations,stud locations, electrical conduit and wiring, vertical plumbing piping,and HVAC systems or other duct work. The acquisition of the data mayallow the model system to accurately locate these interior and exteriorfeatures. Acquisition of As Built data during different points of theconstruction completion allows measurements to be taken prior to aspectsinvolved in a measurement process being concealed by concrete, drywallor other various building materials.

Data is acquired that is descriptive of actual physical features as thefeatures are built and converted into a 3D model which may be referredto as the “As Built” model. The As Built model will include “keycomponents” of the structure and be provided with a level of artificialintelligence that fully describes the key component. In someembodiments, the As Built model may be compared to a design model. Insome implementations “intelligent parameters” are associated with keycomponents within the 3D model. For example, key components andassociated information may further be associated with intelligentparameters. Intelligent parameters for the key components may includethe manufacturer, model number, features, options, operationalparameters, whether or not an option is installed (and if so, itsfeatures and dimensions), any hardware associated with the key component(and its manufacturer and serial number), an owner's manual and servicecontract information, as non-limiting examples. Intelligent parametersassociated with a functional key component such as, HVAC Equipment, mayinclude the manufacturer, model number, capacity, efficiency rating,serial number, warranty start date, motor size, SEER rating, an owner'smanual associated with the equipment, and service contract information.

Key components of the structure may have an identification device suchas a two or three dimensional graphical code (such as a QR code label) aRadio Frequency Identification Chip (RFID) attached that is accessibleto a user, such as a structure owner, structure builder or servicetechnician. When scanned with an apparatus capable of reading the code,a user interface on a display of various types, such as a tablet, mayuse the associated identification, such as a QR code, to provide directaccess to related information. In some examples, the display may showtextual or tabular representations of related data.

In other examples, graphical data such as images, drawings, and the likemay be displayed. In still further examples, both graphical and textualdisplays may be associated with the code. Although a QR code may providean example, other identification technologies such as radio frequencyID, Internet of things (IoT) communication protocols with associatedstored information, and other devices that can receive a signal andrespond with stored information may be used. As well, numerous othertypes of graphical codes in addition to QR code may be read by a deviceand provide a connection between a key component, machinery, locationand other identified aspect and associated data. In some examples, animage based code may be displayed using paints or pigments which are notvisible to the human eye, such as in a non-limiting example ultravioletpigments. In some other examples, a paint or pigment may not be visibleuntil it is made to emit visible light by irradiating it with aparticular band of electromagnetic radiation, such as, for example,ultraviolet light.

In some examples, key components may include doors, windows, masonry,roofing materials, insulation, HVAC equipment and machinery.

An automated Design and Monitoring (“RDM”) system may support dynamicupdating of tracked aspects. For example, as a structure owner acquiresnew or additional key components, such as machinery, HVAC, plumbingadditions, key components may be added into the As Built model and thekey components may be tracked as a part of the model. Other aspects maybe dynamically updated such as when additions are made to the buildingstructure or rebuilding of internal structure is made as non-limitingexamples.

Since the As Built model includes information in a database and dynamicmodel functionality exists that commences as a building structure isbeing constructed, the model may assume new support aspects to theconstruction process itself. For example, a benefit from the definitionand utilization of many components within a Structure utilizing thesystem herein includes the ability to pre-cut and/or pre-fabricate studsand framing, roofing cuts, masonry, under-slab plumbing, HVAC ductwork,electrical, and other such components. The dimensions of these variouscomponents may be dynamically updated based on an original model thatmay be compared to actual fabricated structure as realized on a buildingsite. In some examples a structure builder may use a display interfaceassociated with the system and model to display a comparison of anoriginal set of building plans to a current structure at a point in timewhich may allow the builder to authorize any structural changes orvariances to design and thereafter allow the description of followingcomponents to be dynamically adjusted as appropriate. The system may beof further utility to support various inspections that may occur duringa building project which may associate detected variances with designexpert review and approval. An inspector may be able to utilize thesystem as allowed on site or operate a window into the system from aremote location such as his office.

As the system is utilized during construction, orders for customizedcomponents may be placed. These customized components may be labeled anddelivered to site, in an appropriate sequence, for assembly bycarpenters. This may contribute to a minimization of waste at theworksite, as well as provide a work product that is entirely consistentwith a pre-determined model which may have approved changes that aretracked. The result may improve the quality of the work product and makeit easier to generate the measured point-cloud 3D model.

Performance Tracking

In another aspect, the AVM system can autonomously and/or interactivelyobtain, store and process data that is provided to it by components ofthe Structure as the structure is built, installed or additions are madeto the structure. The generation, modeling, capture, use, and retentionof data relating to Performances in specific equipment or in some casesaspects relating to the design of a facility, may be monitored by thesystem.

In some examples, Operational Performance may be assessed by processingsampled data with algorithms of various kinds. Feedback of the status ofoperation and of the structure as a whole or in part, as assessed byalgorithmic analysis may be made to a structure owner or a structurebuilder. In addition, a variety of data points gathered via appropriateSensors, visual and sound data may be recorded and stored and correlatedto 3D models of the facility. Experiential Sensor readings may include,by way of non-limiting example: temperature, power usage, utilitiesused, consumables, product throughput, equipment settings, and equipmentPerformance measurement, visual and audible data. Techniques to recorddata points may involve the use of one or more of: electronic Sensors,electro-mechanical Sensors, CCD capture devices, automated inspectionequipment, video camera arrays and audio microphones and arrays of audiomicrophones for the capture and processing of data that may be used togenerate visualizations of actual conditions, either on site or at aremote location. In addition, data may be collected, retained, analyzed,and referenced to project facility Performance.

In some examples, data may also be combined with manufacturer equipmentspecifications and historical data to model expectations related toactual operation of the structure and property aspects.

Virtual Maintenance Support

A 3D model of structure, such as a structure, which may be integratedwith information related to the key components and laser scannedlocation information, may be made available to the structureowner/structure builder through a computer, an iPad or tablet, or smartdevice. The resulting system may be useful to support virtualmaintenance support.

The three dimensional model may support enhancement to the twodimensional views that are typical of paper based drawings. Althoughthree dimensional renderings are within the scope of informationdelivered in paper format, a three dimensional electronic model mayrender dynamic views from a three dimensional perspective. In someexamples, the viewing may be performed with viewing apparatus thatallows for a virtual reality viewing.

In some examples, a viewing apparatus, such as a tablet or a virtualreality headset, may include orienting features that allow a user suchas a structure owner, structure builder, inspector, engineer, designeror the like to view aspects of a model based upon a location, adirection, a height and an angle of view. A current view may besupplemented with various other information relating to featurespresented in the view. In some examples, the interface may be accessiblethrough a virtual reality headset, computer, or mobile device (such asan iPad, tablet, or phone), as non-limiting examples. Utilizing a deviceequipped with an accelerometer, such as a virtual reality headset ormobile device, as non-limiting examples, a viewable section of the modelmay be displayed through the viewing medium (whether on a screen, orthrough a viewing lens), where the viewer's perspective changes as theaccelerometer equipped device moves, allowing them to change their viewof the model. The viewer's Vantage Point may also be adjusted, through acertain user input method, or by physical movement of the user, asnon-limiting examples.

The presented view may be supplemented with “hidden information”, whichmay include for example, depictions of features that were scanned beforewalls were installed including pipes, conduits, ductwork and the like.Locations of beams, headers, studs and building structure may bedepicted. In some examples, depiction in a view may include asuperposition of an engineering drawing with a designed location, inother examples images of an actual structure may be superimposed uponthe image based upon As Built scans or other recordations.

In a dynamic sense, display may be used to support viewing ofhypothetical conditions such as rerouted utilities, and rebuild wallsand other such structure. In some examples, graphical or text based datamay be superimposed over an image and be used to indicatespecifications, Performance aspects, or other information not related tolocation, shape and size of features in the image.

As presented above, an image may allow for a user to “see through walls”as the augmented reality viewing device simulates a section of a modelassociated with a space displayed via the virtual reality viewingdevice. The viewer's perspective may change as an accelerometer in thevirtual reality viewing device moves. A user may also change a view ofthe AVM, to include different layers of data available in the AVM. Theviewer's Vantage Point may also be adjusted by moving about a physicalspace that is represented by the model. To achieve this, it may bepossible to incorporate positioning hardware directly into a buildingrepresented by the virtual model. The positioning hardware may interfacewith an augmented reality device for positioning data to accuratelydetermine the viewing device's orientation and location with millimeterprecision. The positioning hardware may include, for example a radiotransmitter associated with a reference position and height. Altitude isdifferentiated from height unless specifically referenced since therelative height is typically more important.

Accordingly, a user may access the AVM on site and hold up a smartdevice, such as an iPad or other tablet, and use the smart device togenerate a view inside a wall in front of which the smart device ispositioned, based upon the AVM and the location, height and direction ofthe smart device position.

In some examples, through the use of an augmented reality device, it mayalso be possible to view data, such as user manuals, etc. of associateddevices in the view of a user, simply by looking at them in the viewinginterface. In other examples, there may be interactive means to selectwhat information is presented on the view.

Various electronic based devices implementing of the present inventionmay also be viewed in a virtual reality environment withoutaccelerometer such as a laptop or personal computer. A viewable sectionof a model may be displayed on a Graphical User Interface (GUI) and theviewer's Vantage Point may be adjusted, through a user input device.

The ability to track machinery and other components of a system andstore the components associated information, such as, for example usermanuals and product specifications and part numbers, may allow for muchmore efficient use and maintenance of the components included within astructure. As well, the system model may also maintain structure ownermanuals and warranties and eliminate the need for storage and trackingof hard copy manuals.

In a non-limiting example, if a structure owner/structure builderdesires information related to a machinery, it may be found bypositioning a device with a location determining device within it inproximity to the machinery and accessing the parallel model in theVirtual Structure such as by clicking on the machinery in the VirtualStructure model or by scanning the Code label attached to machinery. Insome examples, an internet of things equipped machine may have theability to pair with a user's viewing screen and allow the system modelto look up and display various information. Thus, the user may haveaccess to various intelligent parameters associated with that machinerysuch as service records, a manual, service contract information,warranty information, consumables recommended for use such asdetergents, installation related information, power hooked up and thelike.

In some examples, an AVM system may include interfaces of various kindsto components of the system. Sensors and other operational parameterdetection apparatus may provide a routine feedback of information to themodel system. Therefore, by processing the data-stream with variousalgorithms autonomous characterization of operating condition may bemade. Therefore, the AVM system may provide a user with alerts whenanomalies in system Performance are recognized. In some examples,standard structure maintenance requirements may be sensed or trackedbased on usage and/or time and either notification or in some casesscheduling of a service call may be made. In some examples, the alertmay be sent via text, email, or both. The structure user may,accordingly, log back into the Virtual Structure to indicate completionof a maintenance task; or as appropriate a vendor of such service ormaintenance may indicate a nature and completion of work performed.

By detecting operational status, a Virtual Structure may take additionalautonomous steps to support optimal operation of a system. A VirtualStructure may take steps to order and facilitate shipping of anticipatedparts needed for a scheduled maintenance ahead of a scheduled date for amaintenance event (for example, shipping a filter ahead of time so thefilter arrives prior to the date it is scheduled to be changed). Inanother example, a Virtual Structure may recall notes from an OriginalEquipment Manufacturer (OEM) that could be communicated to a userthrough the Virtual Structure. In still further examples, a VirtualStructure may support a user involved in a real estate transaction byquantifying service records and Performance of a real property.

In still another aspect the AVM may establish a standard maintenance andwarranty program based on manufacturers published data and the abilityto advise structure owners of upcoming needs and/or requirements. Inother examples, the model system may facilitate allowing for structurebuilders, rental companies, or maintenance companies to consolidateinformation for volume discounts on parts or maintenance items. Themodel system may also facilitate minimizing unnecessary time expenditurefor structure builders hoping to minimize needless service calls forwarranty issues, and allowing structure builders and rental companiesattempting to sell a structure or a rental to demonstrate that care hasbeen taken to maintain a structure.

Benefits derived from monitoring and tracking maintenance with a VirtualStructure may include positively reassuring and educating lenders and/orlien holders that their investment is being properly cared for. Inaddition, insurance companies may use access to a Virtual Structure toprovide factual support that their risk is properly managed. In someexamples, a data record in a Virtual Structure model system and how anowner has cared for their facility may be used by insurance companies orlenders to ensure that good care is being taken. Maintenance recordsdemonstrating defined criteria may allow insurance companies to offer astructure owner policy discount, such as, for example, installation ofan alarm system. Additionally, access to a Virtual Structure may allowmunicipalities and utilities to use the info for accurate metering ofutility usage without having to manually check; and peaks in utilitydemand may be more accurately anticipated.

In some examples, Virtual Structure may also be used to assist withstructure improvement projects of various types. In some examples, thestructure improvement projects may include support for building largeradditions and modifications, implementing landscaping projects. Smallerprojects may also be assisted, including in a non-limiting example sucha project as hanging a picture, which may be made safer and easier withthe 3D “as-built” point cloud information. Hidden water piping,electrical conduits, wiring, and the like may be located, or virtually“uncovered”, based on the model database.

Optimization of Facilities

During construction of a structure corresponding to a Virtual Structure,discrete features of the As Built structure may be identified via anidentification device such as an IoT device or a QR code label. The IDdevice may be integrated to the feature or added during the build scope.Performance monitors may also be simultaneously installed to allowmonitoring of Key Performance Indicators (KPIs) for selected features.In an example, an HVAC system may be added to a facility duringconstruction and a simultaneously a Performance monitor may be added tothe HVAC system. The Performance monitor may be used to monitor variousKPIs for an HVAC system. These KPIs may include outdoor air temperature,discharge air temperature, discharge air volume, electrical current, andthe like. Similar monitoring capabilities may be installed to allmachinery and utilities systems in a facility. The combination of thesenumerous system monitors may allow for a fuller picture of theefficiency of operations of various systems.

Use of the Virtual Structure, which may include data values contributedfrom communication of data from the various monitoring systems, mayallow owners to receive periodic reports, such as in a non-limitingsense monthly emails which may show their current total energyconsumption as well as a breakdown of what key components arecontributing to the current total energy consumption.

The systems presented herein may be used by owners and facility managersto make decisions that may improve the cost effectiveness of the system.An additional service for Owners may allow the structure owner to tapinto energy saving options as their structure ages. As an example, if amore efficient HVAC system comes on the market, which may includeperhaps a new technology node, the user may receive a “Savings Alert”.Such an alert may provide an estimated energy savings of the recommendedmodification along with an estimate of the cost of the new system. Theseestimates may be used to generate a report to the owner of an estimatedassociated return-on-investment or estimated payback period should thestructure owner elect to replace their HVAC system.

In some examples, an AVM of a Virtual Structure may set a thresholdvalue for the required ROI above which they may be interested inreceiving such an alert with that ROI is achieved. This information willbe based on data derived from actual operating conditions and actualhistorical usage as well as current industry information. Predictivemaintenance and energy savings to key systems via Smart Structure TotalCost of Ownership (“TCO”) branded Sensors.

Aggregating Data from Multiple Residences

With the ability to collect and utilize relevant structure informationwith the model system, the aggregation of data and efficiency experiencefrom numerous systems may allow for analysis of optimization schemes forvarious devices, machinery and other structure components that includesreal installed location experience. Analysis from the aggregated datamay be used to provide feedback to equipment manufacturers, buildingmaterials fabricators and such suppliers.

In some examples, business models may include providing anonymous andaggregated data to original equipment manufacturers as a service modelto give the OEMS an ability to utilize more data to monitor and improvetheir products. In some examples, OEM advertising may be afforded accessthrough the model system. Manufacturers may have an additional sidebenefit motivating the use of this data related to improving theirequipment cost effectives and reliability in order to minimize warrantycost. Such optimized Performance may also provide benefits to bothstructure owners and builders to support their ability to track actualwarranty information, power cost, and overall Performance of astructure.

Methods and Apparatus

Referring to FIGS. 3A-3F, an illustration of the collection of data byscanning a facility during its construction is provided. In FIG. 3A, adepiction of a site for building a facility structure is illustrated.The depiction may represent an image that may be seen from above thesite. Indications of property boundaries such as corners 301 andproperty borders 302 are represented and may be determined based on sitescanning with property markings from site surveys or may be enteredbased on global coordinates for the property lines. An excavatedlocation 303 may be marked out. Roadways, parking and/or loading areas304 may be located. Buried utilities such as buried telephone 305,buried electric 306, buried water and sewer 307 are located in the modelas illustrated. In some examples, such other site service as a buriedsprinkler system 308 may also be located.

Referring to FIG. 3B the excavated location 303 may be scanned or imagedto determine the location of foundation elements. In some non-limitingexamples, a foundational footing 321 along with buried utilities 322 isillustrated. The buried utilities may include such utilities as electriclines, water supply whether from a utility or a well on location, seweror septic system lines, telecommunications lines such as telephone,cable and internet. Other footing elements 323 may be located atstructural requiring locations as they are built. In some examples ascanning system may provide the locational orientation relative to siteorientation markings. In other examples, aerial imagery such as may beobtained with a drone may be used to convert features to accuratelocation imagery.

Referring to FIG. 3C a wall 331 of the Structure in the process of buildis illustrated. The structure may be scanned by a scanning element 330.In some examples, a laser three dimensional scanner may be used. Thewall may have supporting features like top plates 333, headers 336,studs 332, as well as internal items such as pipes 334, electricalconduits and wires 335. There may be numerous other types of featureswithin walls that may be scanned as they occur such as air ducts, datacables, video cables, telephone cables, and the like.

Referring to FIG. 3D the wall may be completed with structure componentsbehind wall facing 340 may no longer be visible. Electrical outlets 341and door structures 342 may be scanned by a scanning element 330.

Referring to FIG. 3E internal components such as machinery may beinstalled. As a non-limiting example, a machine 350 may be installed andthe resulting three dimensional profiles may be scanned by a scanningelement 330. In some examples, an operational monitor 351 may beattached to the machinery. In some examples, an operational monitor maybe part of the machinery. The operational monitor may have the abilityto communicate 352 data to various receivers that may be connected tothe model system of the residence. In some examples, key structuralcomponents, such as doors, may have identifying devices such as a QRlabel 353. The label may be visible or painted into the structure withnon-visible paint. The identifying devices may provide informationrelated to the device itself and warrantees of the device asnon-limiting examples.

The model may include the various structure elements hidden and visibleand may be used to create output to a display system of a user.Referring to FIG. 3F an example display is illustrated. The variousnon-visible layers may be shown by rendering the covering layers with atransparency. Thus, the display shows the machine profile 350 as well asthe internal features that may be concealed like pipes 334, electricalconduits with wires 335, and headers 336 as examples.

Referring to FIG. 3G, an illustration of feedback of the model system isillustrated. A wall that has been scanned with an HVAC unit 360 mayinclude a Performance Monitor 351 which may communication variousinformation wirelessly 352. The communication may be received at anantenna 370 of a router 371 within the facility. The facility may beinterconnected through the internet 372 to a web located server 373which processes the communication. The web located server 373 also caninclude the various model data about the facility and it can providecomposite displays that can summarize the structure as well as theoperational Performance of the HVAC unit 360. It may aggregate thevarious data into textual and graphic reports. In some examples it maycommunicate these reports back through internet connections. In otherexamples, wireless Smart Device communications may be sent to cellulartowers 374 which may transmit 375 to a Smart Device 376 of a userassociated with the facility.

Referring to FIG. 3H an illustration of a virtual reality display inconcert with the present invention is illustrated. A machinery 350 ofthe facility may communicate information to the model server. A user 380may receive may an integrated communication from the server. Theresulting communication may be provided to a virtual reality headset381. The virtual reality headset may provide a display 382 to the userthat provides a three-dimensional view of the physical data as well assimulated imagery that may allow views through objects to hiddenelements behind the object. As well, a heads up type display ofinformation about an object may be superimposed.

Referring now to FIG. 4A, method steps that may be implemented in someembodiments of the present invention are illustrated. At method step401, Deployment aspects may be specified for a Structure andincorporated into a virtual model, such as an AVM discussed above.Deployment aspects may include for example, a purpose for an As Builtstructure that is built based of the AVM. The purpose may include, byway of non-limiting example, one or more of: manufacturing, processing,data processing, health care, research, assembly, shipping andreceiving, prototyping and the like.

Deployment aspects may also include a level of use, such continual,shift schedule or periodic. A climate in which the structure will beplaced may also be considered in the Deployment aspects. Climate mayinclude one or more of: four seasons; primarily winter; tropical,desert; exposed to salt air; and other environmental factors.

At method step 402, a virtual model, such as an AVM is digitally createdaccording to the Deployment aspects of the model. The AVM may includeimprovements to a real estate parcel and a structure that will be placedon the real estate parcel, as well as where a structure may be locatedupon the parcel.

At method step 403, Performance aspects of machinery that may beincluded in the AVM may be digitally modeled and may include a level ofuse of the machinery and an expected satisfaction of the machinery asdeployed according to the Deployment aspects. Maintenance expectations,including a number of repair calls and a preventive maintenance schedulemay also be modeled and associated costs.

At method step 404, Performance aspects of equipment that may beincluded in the AVM may be digitally modeled and may include a level ofuse of the equipment and an expected satisfaction of the machinery asdeployed according to the Deployment aspects. Maintenance expectations,including a number of repair calls and a preventive maintenance schedulemay also be modeled and associated costs.

At method step 405, As Built aspects of a structure are recorded asdiscussed herein, preferably recordation of As Built aspects begins asconstruction begins and continues throughout the existence of thestructure.

At method step 406, the physical structure may be identified via alocation. A physical location may include, for example, CartesianCoordinates, such as Latitude and Longitude coordinates, GPScoordinates, or other verifiable set of location parameters. Inaddition, more exact location specifications may include surveydesignations.

At method step 407, a position within or proximate to the Structure maybe determined via positioning identifiers. The position within orproximate to the Structure may be determined.

At method step 408, an AVM may be identified and accessed via thephysical location. Once an appropriate AVM is accessed, a particularportion of the AVM may be presented via a GUI based upon the positionwithin the Structure (or proximate to the Structure) and a direction,height and angle of view. The position may be determined relative tolocation identifiers. Height may be determined via electronic devices,such as a smart device, or via triangulation referencing the locationidentifiers (locations identifiers are discussed more fully above andbelow).

At method step 409 an update may be made to a physical Structure and atmethod step 410, the update to the physical structure may be recordedand reflected in the AVM.

Referring to FIG. 4B, a method flow diagram for monitoring andmaintenance is illustrated. At 411 a user may obtain a scanning deviceor devices that may scan a building site. At 412, the user or a serviceof the user may mark property boundaries of the site. At 413, work onthe site may continue with the excavation of a building base and thelaying down of utilities and other buried services. At 414, the scanningdevice is used to scan the location of the various aspects of thebuilding site. At 415, work may continue with the laying of footings andfoundations and other such foundational building activities. At 416,scanning of the footings and foundations may be accomplished. At 417, astructure may be framed and features such as pipe conduit, electricalwiring communications wiring and the like may be added. At 418, thebuilding site may again be scanned to locate the various elements. Theframing of the residence may commence along with running of pipe,wiring, conduits, ducts and various other items that are located withinwall structures. Before coverings are placed on walls, the framedstructure may be scanned at 418. Thereafter, the framed structure may beenclosed with walls 419. Thereafter, the walls may be scanned with thescanning device at step 420.

Referring to FIG. 4C a method flow diagram for structure monitoring andmaintenance is illustrated. In this flow diagram, a Structure mayalready be built and may have various data layers already located in themodel system. At 421, machinery may be added to the Structure. At 422,an ID tag, or a QR tag, or and RFID tag or an internet of things devicemay be associated with the machinery and may be programmed into themodel system. At 423, the model system may be interfaced to themachinery ID and into the Structure model. At 424, a scanning step maybe used to input three dimensional structure data at the installedlocation into the model system. At 425, an operational monitor functionof the device may be added or activated. At 426, operational data may betransferred from the operational monitor to the server with theStructure model.

At 427, algorithms running on a server of the model system may determinean operational improvement opportunity based on calculations performedon the data from the operational monitor. At 428 a user may query theoperational data of the machinery for information on its warranty. At429, the model system may initiate an order for a service part and mayschedule a service visit to make a repair based upon analysis of theoperational data. The various steps outlined in the processing flow maybe performed in different orders. In some examples additional steps maybe performed. In some examples, some steps may not be performed.

In some embodiments, the present invention includes a method of trackingattainment of a stated Performance Level relating to a Structure,including: a) determining a geographic position of a Structure via aglobal positioning system device in a smart device proximate to theStructure; b) identifying a digital model of the Structure based uponthe geographic position of the Structure, the digital model comprisingvirtual representation of structural components included in theStructure; c) referencing multiple positioning reference devices withinthe Structure; d) measuring a distance to at least three of the multiplepositioning reference devices from a point of measurement; e)calculating a position within the Structure, the calculation based upona relative distance of the at least three positioning reference devicesto the point of measurement and a triangulation calculation; f)calculating an elevation of the point of measurement; g) measuring afirst state within the Structure with a sensor; h) specifying a locationof the first state within the Structure via reference to the position ofthe point of measurement and the elevation of the point of measurement;i) recording a first time designation for the step of measuring a firststate within the Structure with a sensor; and i) correlating the firststate within the Structure and the first time designation attainment ofthe stated Performance Level.

The geographic position may be calculated with a GPS reading from withinthe Structure. Measuring a distance to the at least three of thepositioning reference devices may include, one or more of: relativesignal strength received from wireless transmissions emanating from theat least three positioning reference devices; time of arrival of radiosignals of wireless transmissions emanating from the at least threepositioning reference devices measuring a distance to the at least threepositioning reference devices comprises time difference of arrival ofradio signals of wireless transmissions emanating from the at leastthree reference positioning devices.

The above steps may be repeated for at least a second state and a secondtime designation, and in preferred embodiments multiple more states andtime designations.

A state may include, for example, one or more of: a vibration measuredwith an accelerometer; a temperature of at least a portion of thestructure; an electrical current measurement to equipment installed inthe Structure, a number of cycles of operation of equipment installed inthe Structure; a number of cycles of operation of an machinery installedin the Structure; an electrical current measurement to an machineryinstalled in the Structure; a vibration associated with movement of anoccupant of the Structure.

A vibration pattern may be associated with a specific occupant andtracking the movement of the specific occupant through the structure maybe based upon measured vibration patterns. Similarly, a vibrationpattern may be associated with a particular activity of a specificoccupant and the activity of the specific occupant may be tracked withinthe structure based upon measured vibration patterns.

A Performance Level may include one or more of: operating the Structurefor a term of years within a threshold use of energy; operating theStructure for a term of years within a threshold number of repairs; andoperating the Structure for a term of years within a threshold budgetarycost.

FIG. 5 illustrates Reference Point Transceivers 501-504 that may bedeployed in a defined area 506, such as a Structure to determine alocation 507 of an Agent 500 within or proximate to the defined area505. Reference Point Transceivers 501-504 may be fixed in a certainlocation and Transceive in a manner suitable for triangulationdetermination a position of an Agent. Transceiving may be via wirelesstransmission to one or more Transceivers 505 supported by the Agent 500.By way of non-limiting example, Transceivers 505 supported by the Agent500 may be included in, or be in logical communication with a smartdevice, such as a smart phone, tablet or other Agent supportable device,such as a headgear, ring, watch, wand, pointer with Transceivers 505able to Transceive with the Reference Point Transceivers 501-504. TheReference Point Transceivers 501-504 may include devices, such as, forexample, a radio transmitter, radio receiver, a light generator, or animage recognizable device. A radio transmitter may include a router orother WiFi, Bluetooth or other communication device for entering intological communication with a controller. In some embodiments, ReferencePoint Transceivers 501-504 may include a WiFi router that additionallyprovides access to a distributed network, such as the Internet.Cartesian Coordinates, X, Y, Z coordinates, vector values, a GPSposition, or other data that may be utilized for one or more of:locating one or both of an Agent 500; indicating a direction ofinterest; and identify a Structure or defined area 506. A radiotransmitter may include a router or other WiFi device. In this way, theProcessing Facility 505 may be profiled with, for example, one or morelocation modalities, such as the aforementioned radio transmitter. Theradio transmitter may include transmissions on traditional WiFifrequencies (300 MHz-60 GHz), including ultra-wideband frequencies(3.5-6.5 GHz). The light beacon may distribute light at human-safeintensities and at virtually any frequency known in the art. Suchfrequencies include, without limitation, infrared, ultraviolet, visible,or nonvisible light. Further, the light beacon may comprise a laser,which may transmit light at any of the aforementioned frequencies but ina coherent beam. This plurality of modalities allows for increasedaccuracy because each modality may have a different degree ofreliability. For example, a smart phone may measure the timing signaltransmitted by a WiFi router within a different error tolerance than itmay measure the receipt into a photodetector of infrared laser light.This has at least two principle benefits. First, the locationcalculation may, in some embodiments, be a weighted average of thelocation calculated from each modality. Second, outliers may be shed.For example, if the standard location calculation comprises a weightedaverage of the location as calculated by five modalities, but onemodality yields a location greater than two standard deviations from theaverage computed location, then that modality may not be considered forfuture weighted location calculations. Additionally, the radiotransmitters and/or transceiver in the smart device may comprisemultiple antennas that transmit signals in a staggered fashion to reducenoise. By way of non-limiting example, if there are three antennas, thenthey may transmit a signal in intervals of 20 ms. Moreover, the antennasmay comprise varying lengths to accommodate desirable wavelengths.Further, as discussed earlier, dead reckoning may be used to measurelocation, using traditional methods of numerical integration. In someembodiments, a position identifier may include a WiFi router thatadditionally provides access to a distributed network, such as theInternet. Cartesian Coordinates, such as a GPS position 506, may beutilized to locate and identify the Structure 505.

By way of non-limiting example, Transceivers 505 supported by the Agent500 may be included in, or be in logical communication with a smartdevice, such as a smart phone, tablet or other Agent supportable device,such as a headgear, ring, watch, wand, pointer with Transceivers 505able to Transceive with the Reference Point Transceivers 501-504. TheReference Point Transceivers 501-504 may include devices, such as, forexample, a radio transmitter, radio receiver, a light generator, or animage recognizable device. A radio transmitter may include a router orother WiFi, Bluetooth or other communication device for entering intological communication with a controller. In some embodiments, ReferencePoint Transceivers 501-504 may include a WiFi router that additionallyprovides access to a distributed network, such as the Internet.Cartesian Coordinates, X, Y, Z coordinates, vector values, a GPSposition, or other data that may be utilized for one or more of:locating one or both of an Agent 500; indicating a direction ofinterest; and identify a Structure or defined area 506.

A precise location may be determined via triangulation based upon ameasured distance from three or more Reference Point Transceivers501-504. For example, a radio transmission or light signal may bemeasured and compared from the three reference position identifiers501-503. Other embodiments may include a device recognizable via imageanalysis and a camera or other Image Capture Device, such as a CCDdevice, may capture an image of three or more Reference PointTransceivers 501-504. Image analysis may recognize the identification ofeach of three or more of the Reference Point Transceivers 501-504 and asize ratio of the respective image captured Reference Point Transceivers501-504 may be utilized to calculate a precise position. Similarly, aheight designation may be made via triangulation using the positionidentifiers as reference to a known height or a reference height.

Referring now to FIG. 5A, according to the present invention, an Agent500 may support one or more Transceivers 506 that include one or bothof: a Multi-modality Transceiver 511 and transceivers of a same modality512; Transceivers of different modalities 513 and a transmitter of asingle modality 514; a transmitter of multiple modalities 515; areceiver of a single modality 516 and a receiver or multiple modalities517. Similarly, a Reference Point Transceiver 501-504 may includemultiple Transceivers, transmitters and receivers 511-518. The multipleTransceivers, transmitters and receivers 511-518 may include one or bothof: transmitters and receivers of a same modality; and transmitters andreceivers of different modalities. FIG. 5A illustrates a genericTransceiver 519 that may be a Reference Point Transceiver 501-504, or anAgent supported Transceiver 505. Both of which may include one or moreof: a Multi-modality Transceiver 511 and transceivers of a same modality512; Transceivers of different modalities 513 and a transmitter of asingle modality 514; a transmitter of multiple modalities 515; areceiver of a single modality 516 and a receiver or multiple modalities517.

A modality, as used in conjunction with a Transceiver, transmitterand/or receiver refers to one or both of a bandwidth of wirelesscommunication and a protocol associated with a bandwidth. By way ofnon-limiting example, a modality, as used in relation to a Transceiver,transmitter and/or receiver may include: WiFi; WiFi RTT; Bluetooth; UWB;Ultrasonic, sonic, infrared; or other logical communication medium.

Referring now again to FIG. 5, according to the present invention,triangulation essentially includes determining an intersection of threedistances 508-510, each distance 508-510 calculated from a referencepoint 501-504 to an Agent supported device 505. The presence inventionallows for a first distance 508, to be determined based upon a wirelesscommunication in a first modality; and a second distance 509 and a thirddistance 510 determined based upon a wireless communication in a same ordifferent modality as the first modality. For example, a first distance508 may be determined based upon a wireless communication using WiFi; asecond distance 509 may be determined based upon a wirelesscommunication using Bluetooth; and a third communication may bedetermined based upon a wireless communication using ultrasoniccommunication (other combinations of same and/or different communicationmodalities are also within the scope of the present invention).

Referring now to FIG. 6 an automated controller is illustrated that maybe used to implement various aspects of the present invention, invarious embodiments, and for various aspects of the present invention,controller 600 may be included in one or more of: a wireless tablet orhandheld device, a server, a rack mounted processor unit. The controllermay be included in one or more of the apparatus described above, such asa Server, and a Network Access Device. The controller 600 includes aprocessor unit 620, such as one or more semiconductor based processors,coupled to a communication device 610 configured to communicate via acommunication network (not shown in FIG. 6). The communication device610 may be used to communicate, for example, with one or more onlinedevices, such as a personal computer, laptop, or a handheld device.

The processor 620 is also in communication with a storage device 630.The storage device 630 may comprise any appropriate information storagedevice, including combinations of magnetic storage devices (e.g.,magnetic tape and hard disk drives), optical storage devices, and/orsemiconductor memory devices such as Random Access Memory (RAM) devicesand Read Only Memory (ROM) devices.

The storage device 630 can store a software program 640 with executablelogic for controlling the processor 620. The processor 620 performsinstructions of the software program 640, and thereby operates inaccordance with the present invention. The processor 620 may also causethe communication device 610 to transmit information, including, in someinstances, control commands to operate apparatus to implement theprocesses described above. The storage device 630 can additionally storerelated data in a database 650 and database 660, as needed.

Referring now to FIG. 7, a block diagram of an exemplary mobile device702. The mobile device 702 comprises an optical capture device 708 tocapture an image and convert it to machine-compatible data, and anoptical path 706, typically a lens, an aperture or an image conduit toconvey the image from the rendered document to the optical capturedevice 708. The optical capture device 708 may incorporate aCharge-Coupled Device (CCD), a Complementary Metal Oxide Semiconductor(CMOS) imaging device, or an optical Sensor 724 of another type.

A microphone 710 and associated circuitry may convert the sound of theenvironment, including spoken words, into machine-compatible signals.Input facilities may exist in the form of buttons, scroll wheels, orother tactile Sensors such as touch-pads. In some embodiments, inputfacilities may include a touchscreen display.

Visual feedback to the user is possible through a visual display,touchscreen display, or indicator lights. Audible feedback 734 may comefrom a loudspeaker or other audio transducer. Tactile feedback may comefrom a vibrate module 736.

A motion Sensor 738 and associated circuitry convert the motion of themobile device 702 into machine-compatible signals. The motion Sensor 738may comprise an accelerometer that may be used to sense measurablephysical acceleration, orientation, vibration, and other movements. Insome embodiments, motion Sensor 738 may include a gyroscope or otherdevice to sense different motions.

A location Sensor 740 and associated circuitry may be used to determinethe location of the device. The location Sensor 740 may detect GlobalPosition System (GPS) radio signals from satellites or may also useassisted GPS where the mobile device may use a cellular network todecrease the time necessary to determine location. In some embodiments,the location Sensor 740 may use radio waves to determine the distancefrom known radio sources such as cellular towers to determine thelocation of the mobile device 702. In some embodiments these radiosignals may be used in addition to GPS.

The mobile device 702 comprises logic 726 to interact with the variousother components, possibly processing the received signals intodifferent formats and/or interpretations. Logic 726 may be operable toread and write data and program instructions stored in associatedstorage or memory 730 such as RAM, ROM, flash, or other suitable memory.It may read a time signal from the clock unit 728. In some embodiments,the mobile device 702 may have an on-board power supply 732. In otherembodiments, the mobile device 702 may be powered from a tetheredconnection to another device, such as a Universal Serial Bus (USB)connection.

The mobile device 702 also includes a network interface 716 tocommunicate data to a network and/or an associated computing device.Network interface 716 may provide two-way data communication. Forexample, network interface 716 may operate according to the internetprotocol. As another example, network interface 716 may be a local areanetwork (LAN) card allowing a data communication connection to acompatible LAN. As another example, network interface 716 may be acellular antenna and associated circuitry which may allow the mobiledevice to communicate over standard wireless data communicationnetworks. In some implementations, network interface 716 may include aUniversal Serial Bus (USB) to supply power or transmit data. In someembodiments other wireless links may also be implemented.

As an example of one use of mobile device 702, a reader may scan somecoded information from a location marker in a facility with the mobiledevice 702. The coded information may include for example a hash code,bar code, RFID or other data storage device. In some embodiments, thescan may include a bit-mapped image via the optical capture device 708.Logic 726 causes the bit-mapped image to be stored in memory 730 with anassociated time-stamp read from the clock unit 728. Logic 726 may alsoperform optical character recognition (OCR) or other post-scanprocessing on the bit-mapped image to convert it to text. Logic 726 mayoptionally extract a signature from the image, for example by performinga convolution-like process to locate repeating occurrences ofcharacters, symbols or objects, and determine the distance or number ofother characters, symbols, or objects between these repeated elements.The reader may then upload the bit-mapped image (or text or othersignature, if post-scan processing has been performed by logic 726) toan associated computer via network interface 716.

As an example of another use of mobile device 702, a reader may capturesome text from an article as an audio file by using microphone 710 as anacoustic capture port. Logic 726 causes audio file to be stored inmemory 730. Logic 726 may also perform voice recognition or otherpost-scan processing on the audio file to convert it to text. As above,the reader may then upload the audio file (or text produced by post-scanprocessing performed by logic 726) to an associated computer via networkinterface 716.

A directional sensor 741 may also be incorporated into the mobile device702. The directional device may be a compass and be based upon amagnetic reading or based upon network settings.

In the following sections, detailed descriptions of examples and methodsof the invention will be given. The description of both preferred andalternative examples though through are exemplary only, and it isunderstood that to those skilled in the art that variations,modifications and alterations may be apparent. It is therefore to beunderstood that the examples do not limit the broadness of the aspectsof the underlying invention as defined by the claims.

Referring now to FIG. 8, exemplary steps that may be performed in someaspects of the present invention are illustrated. At step 801, aprocessor may generate an AVM model of a Structure. The AVM model may bebased upon a physical layout of the Structure and include a layout ofeach item of machinery, equipment as well as facility features. At step802, the AVM may receive data indicative of one or more performancemetrics. Data may include data generated via a sensor and/or input by auser. In some examples, data may include performance metrics, utilitycost, maintenance cost and replacement cost.

At step 803, a data connection between a deployed facility and an AVMmay be automated to generate and transmit data to the model on anautomated basis without human intervention or artificial delay. All orsome data may be stored in a storage. At step 804, the AVM may accessreceived and/or historical data from the same or other AVM models. Atstep 805. Artificial Intelligence routines or other logic may integraterelevant indices, including one or more of: geographic location, labororganization, market conditions, labor costs, physical conditions,property status or data descriptive of other variables.

At step 806, an AVM may generate a value for build and deployment cost,and at step 807 the AVM may include utility and consumables cost. Atstep 808 an AVM may generate one or more of: predicted and actualquantifications from the structure; energy consumption and processthroughput.

Referring now to FIG. 9A, an exemplary perspective graph 900 comprisingthree separate perspective points 925, 945, 965 is illustrated. In someaspects, as illustrated in FIG. 9B, a wearable display 905 may beconfigured to detect eye movement of the wearer 915, which may becalibrated. For example, such as illustrated in FIG. 9B, a neutral,forward-looking eye position 920 may be established as the center pointof the axes 910 (0, 0), which may establish a view along the positivez-axis. As a further illustrative example in FIG. 9C, once calibrated, ashift in eye position 940 to look up and left may change a view from thevantage point and be transmitted to the AVM to access another portion ofthe AVM. As an illustrative example, as shown in FIG. 9D, a user maylook right, and the eye position 960 may shift along the positivex-axis.

In some aspects, the wearable display 905 may comprise a set of gogglesor glasses, wherein the goggles or glasses may comprise one or morelenses. For example, a single wrapped lens may allow a user toexperience panoramic views. Alternately, dual lenses may providedifferent image data, wherein the combined images may allow the user tohave stereoscopic perception of the performance event. In still furtherembodiments, the wearable display 905 may comprise a helmet, which mayallow for more detailed immersion. For example, a helmet may allow fortemperature control, audio isolation, broader perspectives, orcombinations thereof.

Referring now to FIGS. 10A-10C, exemplary horizontal changes in viewingareas are illustrated. In some embodiments, the wearable display maycomprise an accelerometer configured to detect head movement. Similarlyto the eye position detection, the accelerometer may be calibrated tothe natural head movements of a user 1000. In some embodiments, thecalibration may allow the user to tailor the range to the desiredviewing area. For example, a user may be able to move their head 110°comfortably, and the calibration may allow the user to view the entire180° relative the natural 110° movement.

As illustrated in FIG. 10A, a neutral head position 1020 of the wearabledisplay may allow the user 1000 to view a forward-looking perspective1025. As illustrated in FIG. 10B, a right head position 1040 of thewearable display may allow the user 1000 to view a rightward-lookingperspective 1045. As illustrated in FIG. 10C, a left head position 1060of the wearable display may allow the user 1000 to view aleftward-looking perspective 1065.

Referring now to FIGS. 11A-11C, exemplary vertical changes in viewingareas are illustrated. Similarly to FIGS. 10A-10C, in some embodiments,the wearable display may be configured to detect vertical motions. Insome aspects, a user may look up to shift the viewing area to a range inthe positive y axis grids, and user may look down to shift the viewingarea to a range in the negative y axis grids. In some embodiments, thewearable display may be configured to detect both horizontal andvertical head motion, wherein the user may be able to have almost a 270°viewing range.

As illustrated in FIG. 11A, a neutral head position 1120 of the wearabledisplay may allow the user 1100 to view a forward-looking perspective1125. As illustrated in FIG. 11B, an up head position 1140 of thewearable display may allow the user 1100 to view an upward-lookingperspective 1145. As illustrated in FIG. 11C, a down head position 1160of the wearable display may allow the user 1100 to view adownward-looking perspective 1165.

In still further embodiments, the wearable display may be able to detect360° of horizontal movement, wherein the user may completely turn aroundand change the neutral viewing range by 180°. In some aspects, thewearable display may be configured to detect whether the user may besitting or standing, which may shift the perspective and viewing area.In some implementations, a user may be allowed to activate or deactivatethe motion detection levels, based on preference and need. For example,a user may want to shift between sitting and standing throughout theexperience without a shift in perspective. In some implementations, thewearable display may further comprise speakers, wherein audio data maybe directed to the user.

In some embodiments, the wearable display may allow for immersion levelcontrol, wherein a user may adjust the level of light and transparencyof the wearable display and/or frames. In some aspects, the lenses ofthe wearable display may comprise an electrically active layer, whereinthe level of energy may control the opacity. For example, theelectrically active layer may comprise liquid crystal, wherein theenergy level may control the alignment of the liquid crystal. Where auser may prefer a fully immersive viewing experience, the lenses may beblacked out, wherein the user may see the video with minimal externalvisibility. Where a user may still prefer to have awareness orinteractions beyond the video, the lenses and/or frames may allow forsome light to penetrate or may allow for some transparency of the video.

Additional examples may include Sensor arrays, audio capture arrays andcamera arrays with multiple data collection angles that may be complete360 degree camera arrays or directional arrays, for example, in someexamples, a Sensor array (including image capture Sensors) may includeat least 120 degrees of data capture, additional examples include aSensor array with at least 180 degrees of image capture; and still otherexamples include a Sensor array with at least 270 degrees of imagecapture. In various examples, data capture may include Sensors arrangedto capture image data in directions that are planar or oblique inrelation to one another.

Referring now to FIG. 12, methods and devices for determining adirection that may be referenced for one or both of data capture and AVMpresentation of a particular portion of the virtual representation ofthe modeled structure. A User 1200 may position a Smart Device 1205 in afirst position 1201 proximate to a portion of a structure for which arepresentation in the AVM the User 1200 wishes to retrieve and display.The first position 1201 of the Smart Device 1205 may be determined (asdiscussed herein via GPS and/or triangulation) and recorded. The User1200 may then relocate the Smart Device 1205 to a second position 1202in a general direction of the portion of a structure (illustrated as theZ direction) for which a representation in the AVM the User 1200 wishesto retrieve and display. In this manner, the AVM system (not shown inFIG. 12) and/or the Smart Device 1205 may generate one or both of a rayand a vector towards the portion of a structure for which arepresentation in the AVM the User 1200 wishes to retrieve and display.

In some embodiments, the vector may have a length determined by the AVMthat is based upon a length of a next Feature in the AVM located in thedirection of the generated vector. The vector will represent a distance1203 from the second position 1202 to an item 1225 along the Z axisdefined by a line between the first position 1201 and the secondposition 1202. A ray will include a starting point and a direction.

As illustrated, the change in the Z direction is associated with a zerochange in the X and Y directions. The process may also include a secondposition 1205 that has a value other than zero in the X and/or Ydirections.

In other embodiments, a User 1200 may deploy a laser, accelerometer,sound generator or other device to determine a distance from the SmartDevice 1205 to the feature, such as a piece of equipment. Such uniquemethods of determining a location and direction of data capture may beutilized to gather data during construction of modeled buildings orother structures and during Deployment of the structures during theOperational Stage. An additional non-limiting example may includedirection based identification; with a fixed location, or in tandem witha location means, a device may have capabilities to deduce orientationbased information of the device. This orientation information may beused to deduce a direction that the device is pointing in. Thisdirection based information may be used to indicate that the device ispointing to a specific piece of equipment 1225 that may be identified inthe AVM.

In still other embodiments, a device with a controller and anaccelerometer, such as mobile Smart Device 1205, may include a userdisplay that allows a direction to be indicated by movement of thedevice from a determined location acting as a base position towards anAs Built feature in an extended position. In some implementations, theSmart Device determines a first position 1201 based upon triangulationwith the reference points. The process of determination of a positionbased upon triangulation with the reference points may be accomplished,for example via executable software interacting with the controller inthe Smart Device, such as, for example by running an app on the SmartDevices 1205.

In combination with, or in place of directional movement of a SmartDevice 1205 in order to quantify a direction of interest to a user, someembodiments may include an electronic and/or magnetic directionalindicator that may be aligned by a user in a direction of interest.Alignment may include, for example, pointing a specified side of adevice, or pointing an arrow or other symbol displayed upon a userinterface on the device towards a direction of interest.

In a similar fashion, triangulation may be utilized to determine arelative elevation of the Smart Device as compared to a referenceelevation of the reference points.

Other techniques for position determination, such as a fingerprinttechnique that utilizes a relative strength of a radio signal within astructure to determine a geospatial position. are also within the scopeof the present invention.

It should be noted that although a Smart Device is generally operated bya human user, some embodiments of the present invention include acontroller, accelerometer, and data storage medium, Image CaptureDevice, such as a Charge Coupled Device (“CCD”) capture device and/or aninfrared capture device being available in a handheld or unmannedvehicle.

An unmanned vehicle may include for example, an unmanned aerial vehicle(“UAV”) or ground level unit, such as a unit with wheels or tracks formobility and a radio control unit for communication.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or a threedimensional and 4 dimensional (over time) aspect to the captured data.In some implementations, UAV position will be contained within aperimeter and the perimeter will have multiple reference points to helpeach UAV (or other unmanned vehicle) determine a position in relation tostatic features of a building within which it is operating and also inrelation to other unmanned vehicles. Still other aspects includeunmanned vehicles that may not only capture data but also function toperform a task, such as paint a wall, drill a hole, cut along a definedpath, or other function. As stated throughout this disclosure, thecaptured data may be incorporated into an AVM.

In still other embodiments, captured data may be compared to a libraryof stored data using recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Still other aspects mayinclude the use of a compass incorporated into a Smart Device.

By way of non-limiting example, functions of the methods and apparatuspresented herein may include one or more of the following factors thatmay be modeled and/or tracked over a defined period of time, such as,for example, an expected life of a build (such as, 10 years or 20years).

Referring now to FIG. 13, additional apparatus and methods fordetermining a geospatial location and determination of a direction ofinterest may include one or both of an enhanced smart device and a smartdevice in logical communication with wireless position devices1303-1310. The importance of geospatial location and determination of adirection of interest is discussed in considerable detail above. Asillustrated, a smart device 1301 may be in logical communication withone or more wireless position devices 1303-1310 strategically located inrelation to the physical dimensions of the smart device. For example,the smart device 1301 may include a smart phone or tablet device with auser interface surface 1320 that is generally planar. The user interfacesurface 1320 will include a forward edge 1318 and a trailing edge 1319.

In some preferred embodiments, the smart device will be fixedly attachedto a smart receptacle 1302. The smart receptacle 1302 may include anappearance of a passive case, such as the type typically used to protectthe smart device 1301 from a damaging impact. However, according to thepresent invention, the smart receptacle 1302 will include digital and/oranalog logical components, such as wireless position devices 1303-1310.The wireless position devices 1303-1310 include circuitry capable ofreceiving wireless transmissions from multiple wireless positionalreference transceivers 1311-1314. The wireless transmissions willinclude one or both of analog and digital data suitable for calculatinga distance from each respective reference point 1311-1314.

In some embodiments, the smart receptacle 1302 will include a connector1315 for creating an electrical path for carrying one or both ofelectrical power and logic signals between the smart device 1301 and thesmart receptacle 1302. For example, the connector 1315 may include amini-USB connector or a lightening connector. Additional embodiments mayinclude an inductive coil arrangement for transferring power.

Embodiments may also include wireless transmitters and receivers toprovide logical communication between the wireless position devices1303-1310 and the smart device 1301. Logical communication may beaccomplished, for example, via one or more of: Bluetooth, ANT, andinfrared mediums.

Reference transceivers 1311-1314 provide wireless transmissions of datathat may be received by wireless position devices 1303-1310. Thewireless transmissions are utilized to generate a position of therespective wireless position devices 1303-1310 in relation to theAccording to the present invention, reference transceivers 1311-1314providing the wireless transmissions to the wireless position devices1303-1310 are associated with one or more of: a position in a virtualmodel; a geographic position; a geospatial position in a defined area,such as Structure; and a geospatial position within a defined area (suchas, for example a property).

According to the present invention, a smart device may be placed into acase, such as a smart receptacle 1302 that includes two or more wirelessposition devices 1303-1310. The wireless position devices 1303-1310 mayinclude, for example, one or both of: a receiver and a transmitter, inlogical communication with an antenna configured to communicate withreference transceivers 1311-1314. Communications relevant to locationdetermination may include, for example, one or more of: timing signals;SIM information; received signal strength; GPS data; raw radiomeasurements; Cell-ID; round trip time of a signal; phase; and angle ofreceived/transmitted signal; time of arrival of a signal; a timedifference of arrival; and other data useful in determining a location.

The wireless position devices 1303-1310 may be located strategically inthe case 1302 to provide intuitive direction to a user holding the case1302, and also to provide a most accurate determination of direction.Accordingly, a forward wireless position device 1303 may be placed at atop of a smart device case and a rear-ward wireless position device 1304may be placed at a bottom of a smart device case 1302. Some embodimentseach of four corners of a case may include a wireless position device1305, 1306, 1307, 1308. Still other embodiments may include a wirelessposition device 1309 and 1310 on each lateral side.

The present invention provides for determination of a location of two ormore wireless positioning devices 1303-1310 and generation of one ormore directional vectors 1317 and/or rays based upon the relativeposition of the wireless positioning devices 1303-1310. For the sake ofconvenience in this specification, discussion of a vector that does notinclude specific limitations as to a length of the vector and isprimarily concerned with a direction, a ray of unlimited length may alsobe utilized. In some embodiments, multiple directional vectors 1317 aregenerated and a direction of one or more edges, such as a forward edge,is determined based upon the multiple directional vectors 1317.

According to the present invention a geospatial location relative to oneor more known reference points is generated. The geospatial location inspace may be referred to as having an XY position indicating a planardesignation (e.g. a position on a flat floor), and a Z position (e.g. alevel within a structure, such as a second floor) may be generated basedupon indicators of distance from reference points. Indicators ofdistance may include a comparison of timing signals received fromwireless references. A geospatial location may be generated relative tothe reference points. In some embodiments, a geospatial location withreference to a larger geographic area is associated with the referencepoints, however, in many embodiments, the controller will generate ageospatial location relative to the reference point(s) and it is notrelevant where the position is located in relation to a greatergeospatial area.

In some embodiments, a position of a smart device may be ascertained viaone or more of: triangulation; trilateration; and multilateration (MLT)techniques.

A geospatial location based upon triangulation may be generated basedupon a controller receiving a measurement of angles between the positionand known points at either end of a fixed baseline. A point of ageospatial location may be determined based upon generation of atriangle with one known side and two known angles.

A geospatial location based upon trilateration may be generated basedupon a controller receiving wireless indicators of distance and geometryof geometric shapes, such as circles, spheres, triangles and the like.

A geospatial location based upon multilateration may be generated basedcontroller receiving measurement of a difference in distance to tworeference positions, each reference position being associated with aknown location. Wireless signals may be available at one or more of:periodically, within determined timespans and continually. Thedetermination of the difference in distance between two referencepositions provides multiple potential locations at the determineddistance. A controller may be used to generate a plot of potentiallocations. In some embodiments, the potential determinations generallyform a curve. Specific embodiments will generate a hyperbolic curve.

The controller may be programmed to execute code to locate an exactposition along a generated curve, which is used to generate a geospatiallocation. The multilateration thereby receives as input multiplemeasurements of distance to reference points, wherein a secondmeasurement taken to a second set of stations (which may include onestation of a first set of stations) is used to generate a second curve.A point of intersection of the first curve and the second curve is usedto indicate a specific location.

In combination with, or in place of directional movement of a SmartDevice 1301 in order to quantify a direction of interest to a user, someembodiments may include an electronic and/or magnetic directionalindicator that may be aligned by a user in a direction of interest.Alignment may include, for example, pointing a specified side of adevice, or pointing an arrow or other symbol displayed upon a userinterface on the device towards a direction of interest.

In a similar fashion, triangulation may be utilized to determine arelative elevation of the Smart Device as compared to a referenceelevation of the reference points.

It should be noted that although a Smart Device is generally operated bya human user, some embodiments of the present invention include acontroller, accelerometer, and data storage medium, Image CaptureDevice, such as a Charge Coupled Device (“CCD”) capture device and/or aninfrared capture device being available in a handheld or unmannedvehicle.

An unmanned vehicle may include for example, an unmanned aerial vehicle(“UAV”) or an unmanned ground vehicle (“UGV”), such as a unit withwheels or tracks for mobility. A radio control unit may be used totransmit control signals to a UAV and/or a UGV. A radio control unit mayalso receive wireless communications from the unmanned vehicle.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or a threedimensional and 4 dimensional (over time) aspect to the captured data.In some implementations, a UAV position will be contained within aperimeter and the perimeter will have multiple reference points to helpeach UAV (or other unmanned vehicle) determine a position in relation tostatic features of a building within which it is operating and also inrelation to other unmanned vehicles. Still other aspects includeunmanned vehicles that may not only capture data but also function toperform a task, such as paint a wall, drill a hole, cut along a definedpath, or other function. As stated throughout this disclosure, thecaptured data may be incorporated into an AVM.

In still other embodiments, captured data may be compared to a libraryof stored data using recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Still other aspects mayinclude the use of a compass incorporated into a Smart Device.

By way of non-limiting example, functions of the methods and apparatuspresented herein may include one or more of the following factors thatmay be modeled and/or tracked over a defined period of time, such as,for example, an expected life of a build (such as, 10 years or 20years).

Referring now to FIG. 13A, in some embodiments, wireless positiondevices 1303A-1310A may be incorporated into a smart device 1301A andnot require a smart receptacle to house wireless position devices1303-1310. Wireless position devices 1303A-1310A that are incorporatedinto a smart device, such as a smart phone or smart tablet, will includeinternal power and logic connections and therefore not require wirelesscommunication between the controller in the smart device 1301A and theWireless position devices 1303A-1310A.

A person of ordinary skill in the arts will understand that a smartdevice 1301A with integrated wireless position devices 1303-1310 and asmart device 1301 with wireless position devices 1303-1310 in a smartreceptacle 1302 may provide a directional indication, such as adirectional vector 1317 1317A, without needing to move the smart devicefrom a first position to a second position since a directional vectormay be determined from a relative position of a first wireless positiondevices 1303-1310 and a second wireless positional device wirelessposition devices 1303-1310.

In exemplary embodiments, as described herein, the distances may betriangulated based on measurements of WiFi strength at two points. WiFisignal propagates outward as a wave, ideally according to an inversesquare law. Ultimately, the crucial feature of the present inventionrelies on measuring relative distances at two points. In light of thespeed of WiFi waves and real-time computations involved in orienteering,these computations need to be as computationally simple as possible.Thus, depending upon the specific application and means for taking themeasurements, various coordinate systems may be desirable. Inparticular, if the smart device moves only in a planar direction whilethe elevation is constant, or only at an angle relative to the ground,the computation will be simpler.

Accordingly, an exemplary coordinate system is a polar coordinatesystem. One example of a three-dimensional polar coordinate system is aspherical coordinate system. A spherical coordinate system typicallycomprises three coordinates: a radial coordinate, a polar angle, and anazimuthal angle (r, θ, and φ, respectively, though a person of ordinaryskill in the art will understand that θ and φ are occasionally swapped).

By way of non-limiting example, suppose Point 1 is considered the originfor a spherical coordinate system (i.e., the point (0, 0, 0)). Each WiFiemitter e1, e2, e3 can be described as points (r1, θ1, φ1), (r2, θ2,φ2), and (r3, θ3, φ3), respectively. Each of the ri's (1≤i≤3) representthe distance between the WiFi emitter and the WiFi receiver on the smartdevice.

It is understood that in some embodiments, an azimuth may include anangle, such as a horizontal angle determined in an arcuate manner from areference plane or other base direction line, such as an angle formedbetween a reference point or reference direction; and line (ray orvector) such as a ray or vector generated from or continuing to; a smartdevice, or a positional sensor in logical communication with a smartdevice or other controller. In preferred embodiments the ray or vectormay be generally directed from a reference point transceiver towards,and/or intersect one or more of: an item of interest; a point ofinterest; an architectural aspect (such as a wall, beam, header, corner,arch, doorway, window, etc.); an installed component that may act as areference in an AVM (such as for example, an electrical outlet, a lightfixture, a plumbing fixture, an architectural aspect), an item ofequipment, an appliance, a multimedia device, etc.); another referencepoint transceiver or other identifiable destination. Embodiments includea position of the transceiver being determined via use of a polarcoordinate system. The polar coordinate system may include a sphericalcoordinate system

Accordingly, in some embodiments, spherical coordinate system mayinclude reference point transceiver that is capable of determining anangle of departure of a location signal and a transceiver that iscapable of determining an angle or arrival of the location signal; oneor both of which may be used to facilitate determination of anapplicable azimuth.

According to various embodiments of the present invention, one or bothof an angle of departure and an angle of arrival may therefore beregistered by a transceiver that is transmitting and/or receivingwireless signals (e.g. radio frequency, sonic frequency, or lightfrequency).

In some embodiments, orienteering occurs in a multi-story building, inwhich transceivers, (including for example one or more of: WiFitransceivers, UWB transceivers, Bluetooth transceivers, infraredtransceivers and ultrasonic transceivers) may be located above and/orbelow the technician. In these embodiments, a cylindrical coordinatesystem may be more appropriate. A cylindrical coordinate systemtypically comprises three coordinates: a radial coordinate, an angularcoordinate, and an elevation (r, θ, and z, respectively). A cylindricalcoordinate system may be desirable where, for example, all WiFi emittershave the same elevation.

Referring now to FIG. 13B, in some embodiments, one or both of a smartdevice 1301 and a smart receptacle 1302 may be rotated in a manner (suchas, for example in a clockwise or counterclockwise movement 1320 1322relative to a display screen) that repositions one or more wirelessposition devices 1303-1310 from a first position to a second position. Avector 1326 may be generated at an angle that is perpendicular 1325 orsome other designated angle in relation to the smart device 1301. Insome embodiments, an angle in relation to the smart device isperpendicular 1325 and thereby viewable via a forward looking camera onthe smart device.

A user may position the smart device 1301 such that an object in adirection of interest is within in the camera view. The smart device maythen be moved to reposition one or more of the wireless position devices1303-1310 from a first position to a second position and thereby capturethe direction of interest via a generation of a vector in the directionof interest.

Referring now to FIG. 13C, as illustrated, a vector 1325 indicative of adirection of interest 1325 may be based upon a rocking motion 1323-1324of the smart device 1301, such as a movement of an upper edge 1318 in aforward arcuate movement 1323. The lower edge 1319 may also be moved ina complementary arcuate movement 1324 or remain stationary. The movementof one or both the upper edge 1318-1319 also results in movement of oneor more wireless position devices 1303-1310. The movement of thewireless position devices 1303-1310 will be a sufficient distance toregister to geospatial positions based upon wireless transmissions. Arequired distance will be contingent upon a type of wirelesstransmission referenced to calculate the movement. For example, aninfrared beam may require less distance than a WiFi signal, and a WiFitransmission may require less distance than a cell tower transmissionwhich in turn may require less distance than a GPS signal. In someembodiments, as discussed further below, hybrid triangulation mayinclude one or more distances based upon wireless transmissions ofdifferent bandwidths or modalities. For example, a first modality mayinclude WiFi transmissions and a second modality may include Bluetoothtransmissions, still another modality may include infrared or ultrasonicmodalities.

Referring to FIG. 13D, line segments 1331-1338 are illustrated thatintersect various generated position points (PP1-PP8) for Transceivers1303-1310. Position points PP1-PP8 may be generated according to themethods and apparatus presented herein, including a mathematicalaverage, median, mean or other calculation of multiple positionsdetermined via triangulation techniques. In addition, a vector 1339 orray may be generated based upon one or more of the lines 1331-1338. Insome embodiments, position points may be recorded in high numbers basedupon thousands of logical communications per second and a virtualrepresentation of the position points PP1-PP8 may be generated basedupon the recorded position points PP1-PP8. Some embodiments may alsoinclude a cloud point type representation a device that comprises theTransceivers used to record position point PP1-PP8, wherein the cloudpoint representation is based upon the multiple positions calculated.

Referring now to FIG. 14, in still other embodiments, a smart device1415 may be logically associated with a larger platform 1400 forsupporting wireless position devices 1401-1412. The larger platform 1400may include a vehicle, such as an automobile, a truck, a ship, anaircraft, a motorcycle or other motorized vehicle. As illustrated theplatform 1400 includes an automobile. The platform 1400 may includealmost any combination of two or more wireless position devices1401-1412 that may provide respective positional data sufficient togenerate a directional vector. Accordingly, by way of non-limitingexample, a front and center wireless position device 1401 may be pairedwith a rear center wireless position device 1402; each corner of thevehicle may include a wireless position device 1403-1406; interiorcorners may include a respective wireless position device 1409-1412; andexterior locations, such as on rear view mirrors may contain wirelessposition devices 1407-1408.

Utilizing multiple on board wireless position devices 1401-1412, it ispossible to ascertain a direction that a vehicle is pointing withoutmovement of the vehicle. This is useful since unlike traditional methodsutilized by navigational systems that relied on a first geographiclocation of the vehicle and a second geographic position of the vehicle,which in turn required motion, the present invention provides fordirectional orientation without movement of the vehicle.

In another aspect, a controller may be included in a smart device pairedto the vehicle and/or a transmitter 1416 may transmit data received fromthe multiple wireless position devices 1401-1412 to a remote processorwhich may determine a directional orientation. The remote processorand/or a smart device may also transmit the directional orientation backto a display viewable by an operator of the vehicle.

Referring now to FIGS. 15A-15C, a support 1500 for a smart device 1501is illustrated. The support remains stationary in relation to a groundplane. One or more position devices 1503-1508 are shown located within,on or proximate to the smart device 1501. In FIG. 15A, generally linearmovement 1514-1515 from a first position to a second position isillustrated. In FIG. 15B the extended position 1514B along the generalmovement is illustrated. In some embodiments, a cessation of movement ina general direction is determined via an accelerometer included in oroperated by the smart device 1501. In exemplary embodiments 15A-15C auser (shown here as the support 1500) may activate a user interactivedevice, 1502 such as a button on a touch screen, or a switch to indicateone or both of the first position and the second position.

The wireless position devices 1503-1508 enter into logical communicationwith multiple wireless positional reference transceivers 1510-1513.

In some embodiments, a direction of interest will include an item ofinterest 1509, such as an apparatus or other piece of equipment. Adirection of interest 1514 may include a vector with a directionpointing towards the item of interest 1509. The vector length will besufficient to reach the item of interest 1509.

In some embodiments, a vector indicating a direction of interest 1514may be used to reference an AVM and the SVM may provide a selectionmechanism, such as a drop down menu that includes potential items ofinterest 1509 along the vector direction. A selection of an item ofinterest may then be used to determine a length of the vector 1514.

Referring now to FIG. 15C, a movement of a smart device 1501 may bearcuate in nature 1514C so long as arcuate movement 1514C results insufficient distance of movement of one or more position devices 1503,1505-1508.

Referring now to FIG. 16, method steps that may be implemented in someembodiments of the present invention are illustrated. At method step1600, geospatial location services are used to determine geospatiallocation such as a location of the structure with a position anddirection of interest. Geospatial services may be used to determine auser's location relative to the structure and directions thereto. Themethods used may include, by way of non-limiting example, one or moreof: satellite-based global positioning systems (GPS), cell towertriangulation, radio signal triangulation, Wi-Fi signal locationservices, infrared transmitters and the like.

Geospatial location services will be cross-referenced with databaseregistry of as built virtually modeled facilities and may be used inconjunction with a network of registered service technicians to routethe nearest available service technician to the structure experiencingequipment malfunction. Service technician may register with the systemto accept geospatial location tracking services by the system.

At method step 1601, the service technician's entry into the structurewill be registered. Registration of entry into the structure may beachieved through multiple methods, which may include, by way ofnon-limiting example, on or more of: WiFi gateway detection, infrareddetection, magnetic door locking systems, Bluetooth services, and thelike. Upon entry into the structure requesting the service call, systemwill register the service technician's entry into the structure.

At method step 1602, a support unit for a smart device, such as servicetechnician or an unmanned vehicle may be tacked via a change intriangulation values and/or an accelerometer and a position anddirection within the structure is tracked. The methods used may be, bymeans of non-limiting example, one or more of: use of data gleaned fromaccelerometers located on or in possession of service technicians, WiFiservices, radio frequency (RF) triangulation, Bluetooth technology,infrared detection, RFID badges, and the like.

At method step 1603, a smart device will be registered as enteringwithin structure. Following the smart device entry into structure.

At method step 1604, a smart device may be associated with one or bothof a person and an entity.

At method step 1605, the smart device is pre-registered by the systemwith detailed instructions regarding a reason for the device to be at aparticular location. The reason may be, for example, one or more of: aservice call placed from structure to system detailing current equipmentmalfunction, service calls from structure detailing non-specificmalfunctions and symptomatic data indicating equipment malfunction, aservice call placed by self-assessing equipment utilizing internet ofthings (IoT) and machine learning functionality to ascertainmalfunctions and predictive analytics to anticipate malfunctions, andthe like. The system may integrate data reports into the AVM and relayas much to the smart device in the field.

Alternatively, at method step 1605A, the smart device may arrive at thestructure without prior knowledge of a purpose. Upon entry into thestructure and registration of the smart device as described in methodsteps 1601 through 1604, system will relay data gleaned from the AVM,operational data uploaded to the system through IoT processes, and otherexperiential data reported to the system and thereby relayed to thesmart device on site. Methods for relation of such data to the onsitesmart device may include, by means of non-limiting example, referentialdata based on proprietary orienteering processes to determine smartdevice location within structure, which location will becross-referenced with AVM data.

At method step 1606, a position within or proximate to the structure maybe determined via positioning identifiers. The position within orproximate to the structure is determined and detailed instructionsdirecting smart device to the source of a malfunction is relayed by thesystem to the smart device directly or by means of smart deviceapplication. The methods used may be, by means of non-limiting example,one or more of: augmented reality overlays displayed on heads-updisplays or other wearable technologies, augmented reality overlaysdisplayed on smart devices, direct instructional vectoring provided tothe smart device by the system over WiFi internet connection or LTEsignal, virtual reality walkthrough instructions provided to smartdevice on site or prior to arrival at the structure, updatedmap/schematic displays detailing the structure and directing the smartdevice to the source of the subject malfunction by means of vectoringand orienteering processes.

At method step 1607, a smart device's location within the structurealong an XY axis will be tracked and recorded by the system by means offixed or adaptive orienteering apparatus within the structure. Suchorienteering apparatus may include, by means of non-limiting example, onor more of: WiFi triangulation, infrared position detection, radiofrequency (RF) detection, RF ID tracking, onboard accelerometers locatedon the smart device or carried smart devices, and the like.

At method step 1608, the smart device's location within the structurealong the Z axis will be determined. The methods used may be, by meansof non-limiting example, one or more of: onboard magnetometers, onboardbarometers, onboard accelerometers, and the like, used in conjunctionwith in-structure XY axis position processes described in method step1607 above, along with data detailed in the AVM of the structure.

At method step 1609, the smart device's direction of interest will bedetermined. Method steps 1601 through 1608 work in conjunction to trackand direct the smart device to the source of the malfunction; once atthe source of the malfunction, smart device will be oriented to thedirection of interest. The system will determine the smart device'sdirection of interest using, by means of non-limiting example, on ormore of the following methods: infrared pointers, laser directionfinding devices, onboard camera(s), RF ID trackers, RFD finders, barcodescanners, hex/hash code scanners, WiFi triangulation, and the like.

At method step 1610, the smart device's distance to the subjectmalfunction will be determined. The methods used may be, by means ofnon-limiting example, one or more of the following: infrared pointers,laser pointing devices, WiFi triangulation, RF ID sensors, RFD, depthperception sensors contained within onboard cameras, onboardmagnetometers, Bluetooth technology, ANT sensors, directionally enabledsmart device cases, and the like.

At method step 1611, records of equipment and/or area of interest willbe accessed and relayed to smart device. The smart device's position,direction of interest, and distance to the equipment/area of interest asdetermined by method steps 1601 through 1610 will be cross-referencedwith the AVM and experiential data to call up pertinent data on themalfunctioning equipment/area of interest. Data regarding the servicecall will be added to the AVM and experiential data displayed to theon-site smart device. The methods used may be, by means of non-limitingexample, one or more of: IoT data relayed by machine learning-enabledequipment, structure-relayed symptomatic data, and the like.

Proceeding to FIG. 16A, at method step 1612, symptomatic malfunctiondata will be diagnosed to determine cause of malfunction. The methodsused may be, by means of non-limiting example, one or more of: IoTexperiential data gathered and collated from multiple sources acrossmultiple facilities similar to the presented symptomatic data,internet-gathered data analyzed by various machine learningtechnologies, algorithmic analytics of symptomatic data to determinecausal indications, and smart device expertise.

At method step 1613, technical maintenance data, records, andinstructional walkthrough data will be relayed to smart device. Systemwill collate data from method step 1612 above and relay as much to smartdevice. The methods used may be, by means of non-limiting example, oneor more of: augmented reality overlays as displayed by heads-up displaysand other wearable technologies, augmented reality overlays as displayedon smart devices, virtual reality walkthroughs as shown by wearabletechnologies or smart devices, direct instruction or remote control,.pdf user manuals and other written instructional material, videowalkthrough instructions displayed on smart devices, and the like.

At method step 1614, results of purpose for a presence at a location arerecorded and added as experiential data to the AVM. The methods used maybe, by means of non-limiting example, on or more of: equipmentself-reporting through IoT and machine learning technologies, smartdevice entered data, experiential data gathered from emplaced sensorsand other recording devices within the structure itself, and the like.

Referring now to FIG. 17A, methods steps that may be executed in someembodiments of the present invention are presented. At step 1701, asdiscussed in detail herein, Transceivers may be affixed to referencepositions within or proximate to a structure. In some preferredembodiment's Transceivers are positioned at the perimeter of thestructure or within a defined area of a property and are capable ofwireless communication of logical data within the structure and/ordefined area.

At step 1702, a sensor is deployed to a position within, or proximateto, the structure and/or defined area in a manner conducive for thesensor to operate and generate a logical communication including digitaldata descriptive of a condition that is one or both of: within thestructure and/or defined area, or proximate to the structure and/ordefined area. The sensor will also generate a digital signal descriptiveof the condition monitored by the sensor.

At step 1703, the sensor is activated to an operational state andgenerates the digital data descriptive of a condition and transmits thedigital data descriptive of a condition in the form of a logicalcommunication. In some embodiments, the sensor will transmit the datavia a wireless transmission. In other embodiments, the sensor maytransmit the data via an electrical or optical connector. Sensors mayalso be incorporated into, or in logical communication with a sensorcluster platform capably of operating multiple sensors simultaneously.

At step 1704, a physical position of the sensor is determined. Thephysical position may include a location within the structure and/ordefined area. The physical position may be based upon one or more of: aphysical connection to an item of known location (such as a sensorcluster or an electrical outlet); and wireless communication of thesensor with two or more of the wireless transceivers at the referencepositions. As discussed herein, the physical position may include an Xcoordinate and a Y coordinate on an X, Y plane and an elevation basedupon a Z coordinate relative to a ground plane or other designated planeof origin.

At step 1705 a digital signal is transmitted descriptive of thecondition of the structure. The condition of the structure may be basedupon Vital Conditions of the structure assessed via the sensor readings.At 1706 a physical state of the building at the physical position of thesensor, or an area of the structure within range of the sensor, may becorrelated with the digital signal descriptive of a condition of thestructure.

At step 1707, the sensor locations and/or areas of the structure withinrange of the sensor for which the sensor may take a reading, areassociated with location coordinates, such as X, Y and Z coordinates.

A step 1708, in another aspect, a direction of the sensor in relation toan Agent supported position Transceiver home may be determined via theprocesses described herein. At step 1709 a distance of the sensor to anitem of equipment or an area of interest may be determined. The distancemay be determined via methods, such as calculation within a virtualmodel, infrared reflection, sonic transmissions, LIDAR, echo, User entryor other method.

At step 1710, an index may be activated based upon a time at which asensor was activated. In some preferred embodiments, the index includesentries from a timing device (such as a clock) and also includes achronological sequence of sensor readings and time of reading. The indexmay also be used to synchronize multiple sensor readings, such assynchronization to the clock and thereby capture a holistic picture of astructure during a given time period.

The logical communication may include wireless communications via anindustrial scientific and medical (ISM) band wavelength which mayinclude wavelengths between 6.765 MHz and 246 GHz. WiFi is one protocolthat may be implemented for wireless communication, as is Bluetooth,ANT, infrared or other protocol. Other wavelength bandwidths may includesonic wavelengths and infrared wavelengths.

Referring now to FIG. 17B, method steps for some deployments ofaccelerometers and/or other sensors are presented. At step 1711 avibration is introduced into a component of the structure. The vibrationmay be introduced via a calibrated vibration producing device, or via anoccurrence within or proximate to the sensor. By way of non-limitingexample, a vibration inducing device may generate a vibration patternthat may be tracked through at least a portion of a structure.Nonlimiting examples of an occurrence may include, one or more of:starting and/or running of a machine; movement of an Agent within thestructure; and operation of a vehicle outside of a structure.

At step 1712 the vibration pattern introduced may be compared with apattern of vibration detected by a sensor, such as a MEMS. At step 1713a series of MEMS or other sensor readings may be tracked and at step1714 the pattern of vibrations measured by the MEMS accelerometers maybe correlated with structural integrity, wherein structural integritymay include an assessment of whether a structure is suitable for aparticular use (such as occupation as a residence, commercial use orability to bear a particular load.

At step 1715, structural damage may be correlated with a pattern ofvibration measured. At step 1716, a threshold range of values measuredby a sensor may be set and an alert routine may be executed in the eventthat the threshold range is exceeded or upon an analysis that detects aparticular signal indicating a condition in the structure.

Localization and Direction Orientation

In some exemplary embodiments of the present invention, one or both ofradio frequency and sound wave transmission may be used to determine alocation of a smart device an Agent indicated direction of interest,such as point to an element of interest. In some examples, thislocalization and direction determination may be classified as sixdegrees of freedom of orientation. Referring to FIG. 18A, anillustration of the six degrees of freedom are illustrated. In FIG. 18A,a set of sensing and/or transmitting devices within a defined area 1800are illustrated at a first point 1801, and at a second point 1802. Thepositional coordinates of the first point 1801 may be represented in aCartesian Coordinate system as P_(x), P_(y) and P_(z). Accordingly, thesecond point 1802 may be represented in a Cartesian Coordinate systemP′_(x), P′_(y) and P′_(z). A direction from the first point 1801 to thesecond point 1802 may be represented by a vector 1803 as V_(x), V_(y)and V_(z). An agent 1804 may be located at the first point 1801 in anexample. In representing the orientation and direction of an element ofinterest there may be a number of possible position references that maybe associated with the element.

In some examples, a controller determining a position may default toeither a first point 1801 or a second point 1802 (additional points mayalso be calculated and used according to the methods presented in thisdisclosure) or a mathematical combination of the first point, secondpoint (and any additional points) locations. A vector 1803 may begenerated and a direction of the vector may be used as an Agent defineddirection of interest.

A hierarchy of the first point 1801 to the second point 1802 may bespecified to generate a starting point of the vector (e.g. first point1801) and an intersecting point (e.g. second point 1802), a magnitudemay be generated based upon a model of a position of the Agent. Agenerated direction may be inverted by swapping the hierarchy of thefirst point 1801 and the second point 1802.

One or more of radio frequency and sound frequency transmissions,emissions, reflections, absorption and detections may be used as inputinto a controller for determining a location of the first point 1801 andthe second point 1802 and generation of a direction of interest.

Referring now to FIG. 18B, a defined area 1800 may be equipped withfixed reference point transceivers 1810-1813, each transceiver capableof one or both of transmitting and receiving one or both ofradiofrequency encoded data and soundwave encoded data. Numerousfrequency bandwidths are within the scope of the invention, includingradio waves that are sometimes referred to as Ultra-wideband (UWB)technology which focuses radio wave emissions of low power consumptionto achieve high bandwidth connections, WiFi bandwidths, including WiFiRTT and frequencies compliant with 802.11 specifications, ultrasonicbandwidths, infrared bandwidths and Bluetooth bandwidths, includingBluetooth 5.1.

In some embodiments, each transceiver 1810-1813 may in turn includemultiple transmitters and/or receivers 1814-1817. The multipletransmitters and receivers 1814-1817 may operate on a same or differentfrequencies. Different frequencies may be within a same bandwidth, suchas for example UWB bandwidth, or the different frequencies may be acrossdifferent bandwidths, such as, for example an UWB and a WiFi bandwidth.In some embodiments a single transceiver 1810-1813 may thereby operateon multiple different frequencies. In other embodiments, differenttransceivers 1810-1813 may operate on a same or different frequencies.The multiple transceivers 1810-1813 may be operative to implementsimultaneous or sequenced transmitting and receiving.

In some embodiments, some or all of the multiple transmitters andreceivers 1814-1817 may be incorporated into a transceiver device. Themultiple transmitters and receivers 1814-1817 may also include antennawith a same or different physical characteristics. For example,different antenna may be tuned to a same or different frequencies. Insome embodiments, tens, hundreds or more antenna may be incorporatedinto a device in order to enable redundant communications and improvequality of a wireless communication.

Wireless communication may be accomplished for example via bandwidthsassociated with one or more of: Bluetooth; UWB; WiFi (including RTTWi-Fi); Ultrasonic; and infrared communications. Transceivers used intransceiving may include directional and/or omni-directional antennas1820. Antennae 1820 may be tuned similarly or tuned differently.Transceiving may be accomplished simultaneously, in timed sequenceand/or based upon occurrence of an even. For example, a sensor mayTransceive on a predetermined timed schedule and also Transceivefollowing the occurrence of an event, such as a sensor reading thatexceeds a threshold.

As illustrated in FIG. 18B, at least three Reference Point Transceivers1810-1813 are mounted in different reference locations within orproximate to the defined area 1800. Preferably each Reference PointTransceivers 1810-1813 has a relatively clear line of sight to an AgentTransceiver 1818-1819 supported by an Agent (not shown in FIG. 18B) andthe line of sight is conducive to successful wireless communications.

In some examples, mounting (either permanent or temporary) will includefixedly attaching a Reference Point Transceiver 1810-1813 to a positionand may be made to one or more of: a ceiling within the defined area, awall mount; a stand mount; a pole; or integrated into or otherwiseconnected to an electrical receptacle. In some examples, a referencetransceiver 1810-1813 may be mounted at a calibrated location within thedefined area 1800 and act as a coordinate reference location.

The Reference Point Transceivers 1810-1813 may be placed in logicalcommunication with a controller (such as via a distributedcommunications system, wireless or hardwired), the controller maycyclically receive logical communication from one or more transceiverssupported by an Agent located within the defined area 1800 whilesimultaneously monitoring the reference location. A Reference PointTransceiver 1810-1813 may be useful for calibrating various aspects ofwireless communication between a Reference Point Transceiver 1810-1813and an Agent supported Transceiver 1818-1819, aspects may include, forexample variables in communication relating to one or more of:environmental condition such as humidity, temperature and the like; aswell as a variation in transceiver power levels, noise levels,amplification aspects and the like.

There may be numerous sources and causes of noise in a radiofrequencyenvironment and/or a sound frequency environment (such as ultrasonic)that may come into play when using a Reference Point Transceivers1810-1813 and an Agent supported Transceiver 1818-1819 that operate inone or more of: WiFi bandwidth; Bluetooth bandwidth; Ultra-wideband;ultrasonic or similar technology. For example, in an indoor environmentwalls, structures, furniture, occupants, HVAC settings; particulate inthe air (such as smoke or steam) human traffic; machinery movement; andthe like may create a complex and dynamic environment whereradiofrequency logical communications reflect and are absorbed.Reflections, particularly multiple reflections, may cause spuriouslogical communications where the time for the logical communicationtransmission may be inappropriately long.

Accordingly, in some embodiments, a controller may benefit fromreceiving many data from multiple closely sequenced logicalcommunications included in transmissions/receptions between ReferencePoint Transceivers and Transceivers supported by an Agent. Examples ofmultiple logical communications include less than ten samples tobillions of samples per second. A large number of logical communicationsmay be averaged or otherwise mathematically processed to determine alocalization. Mathematical processing may include less consideration(e.g. weight, or removal) of logical communications outside of anotherwise closely associated data set. Other mathematical processing mayinclude a mean, an average and a median of data included in the logicalcommunications.

Systems with Transceiver counts of as few as six and samplingfrequencies in the hundreds of billions of samples per second have beendemonstrated to localize Transceiver locations with sub-millimeteraccuracy. High sampling rates may require specialized data acquisitioncapabilities including advanced filtering systems, ultrafast digital toanalog converters and the like. Fundamentally, the more samples that arecollected per unit of time a more accurate a position determination maybe.

A wireless positioning system (such as, WiFi, UWB, Ultrasonic andBluetooth) with high positioning accuracy may be used for determinationof a direction of interest using transceivers sized to be unobtrusive ina defined area 1800 and/or to be able to be supported by a human Agentor an automation Agent capable of traversing the defined area 1800.

Referring now to FIGS. 18C-18E, an example of Agent supportedTransceivers 1818-1819 may include a combination of an Agent's smartphone 1861 and an ancillary position determining device 1860, 1870-1871,1881-1882 linked to the smart phone 1861. An ancillary positiondetermining device 1860, 1870-1871, 1881-1882 may provide one locationposition, such as for example, a first location position (P1), and thesmart phone 1861 may provide another location position, such as forexample a second location position (P2). A vector may be generated basedupon P1 and P2. For example, a generated vector may intersect P1 and P2,or the vector may be in a calculated direction, such as an angle, withreference to one or both of P1 and P2.

Linking between a smart device, such as a smart phone 1861, and anancillary position determining device 1860, 1870-1871, 1881-1882 may beaccomplished, for example via a hardwire connection, such as alightening port or USB, mini USB type connector, Bluetooth, ANT, nearfield communications and the like. A smart wrist watch 1860 that may beworn on an Agent's arm, a wand 1883 may be held in an Agent's hand,similarly, a ring 1870 may be worn on a finger and a tag 1871 may beincorporated into a badge, a button, an adhesive backed patch, a clip ora wide variety of attachment mechanisms. Each ancillary positiondetermining device 1860, 1870-1871, 1881-1882 may include one or moreTransceivers capable of communicating with a Reference Point Transceiverto generate logical communications from which a position may becalculated.

The Agent's smart phone 1861 and an ancillary position determiningdevice 1860, 1870-1871, 1881-1882, may each have one or moreTransceivers and may be used with the methods and apparatus describedherein to determine a first point and a second point. The first pointand the second point may be used for generating a vector indicating adirection of interest (as discussed above). Other combinations ofdevices may be used, such as those illustrated in FIG. 18D where a smartring 1870 and a smart tag 1871 may be used to determine the multipleposition location samples.

Referring to FIG. 18E in some embodiments, a single an ancillaryposition determining device 1860, 1870-1871, 1880-1881 may be able toemploy multiple transceivers on its body. For example, a wand 1880 mayinclude a tip Transceiver 1881 and a base Transceiver 1882. A wand 1880may be used in conjunction with a smart device, such as a smart phone1861, where the phone 1861 is in a first position in close proximity toan Agent (such as in a pocket or holster worn by the Agent). The wand1880 may be extended out from a handle portion of the wand

The devices shown as examples may allow a single hand to be used toindicate position and direction. Various other devices that may includetransceiver capability may be used in similar manners. A user may haveone hand occupied holding a tool or sensor or may be otherwise occupiedand can still indicate a desired direction of focus. In the example of awand 1880, the user may press a button, switch, or engage otheractivation mechanism, such as a capacitive discharge device on the wandto indicate that the orientation of the wand is at a desired inputcondition.

Transceiver devices may be operative to employ various methods toimprove accuracy of location determination, including but not limitedto: varying a frequency of transmission and reception of logicalcommunications; varying a pulse pattern transmission and reception oflogical communications, and varying intensity of emissions used intransmitting a logical communication.

In some embodiments of the present invention, Agent supportedTransceivers 1860, 1870-1871, 1881-1882 may communicate bursts oflogical communications that include timing information. A delay oflogical communications between the transmitter and the receiver may beconverted to a distance measurement and a combination of a number ofsuch logical communications may be used to triangulate the position. Insome examples, the smart phone 1861 and an ancillary positiondetermining device 1860, 1870-1871, 1881-1882 may transmit the timinglogical communications which the mounted transceivers receive andprocess for a distance determination. In other examples, an ancillaryposition determining device 1860, 1870-1871, 1881-1882 and smart phone1861 may receive logical communications and determine timing delays andassociated distances. Results of distance determinations may becommunicated to controller, such as processing devices located at asmart device. A suitable controller may be located at one or more of theTransceivers or at a location remote to the Transceivers and connectedby a communication network.

There may be many physical properties that may be used to makelocalization measurements/determinations. In an example of another typeof sensing system an Infrared based sensor and camera system may be usedto determine localization and orientation.

Referring to FIG. 19A, in some embodiments, one or several wirelesscommunications modalities may be operational during a same time period.For example, one or more of UWB; Bluetooth; WiFi; ultrasonic andinfrared transmitters may be included in a system in a defined area1900. The system may include three Reference Point Transceivers1910-1912 that are positioned to transmit to a portion 1940 of thedefined area 1900. Some Reference Point Transceivers operate withoutmovement of the Reference Point Transceivers 1910-1912. Additionalembodiments may include one or more of the Reference Point Transceivers1910-1912 sweeping the defined area 1900, or otherwise changing a fieldof view associated with the respective Reference Point Transceivers1910-1912. For systems that include Reference Point Transceivers1910-1912 that change a field of view, a timing sequence may begenerated and used to correlate with a logical communication such thatthe logical communication is associated with both the Reference PointTransceivers 1910-1912 and a particular field of view.

Some particular embodiments will include Reference Point Transceivers1910-1912 that include one or more infrared cameras, each camera willhave a defined field of view. Other directional transceivers may operatesimilarly.

A Transceiver may be located at position 1915 and wirelessly communicatewith a multitude of the Reference Point Transceivers 1910-1912. Asmentioned in reference to FIGS. 17A-17C a user may wear one or moreTransceivers that include transmitters 1913, such as infrared emittingLEDs, laser or lights that emanate logical communications; WiFi, UWB;Bluetooth and/or Ultrasonic Transceivers. One or more of the ReferencePoint Transceivers 1910-1912 receive logical communications transmittedvia an Agent supported Transceiver 1913. The infrared transmitters 1913may change intensity, cycle on and off, and/or vary in patterns toenable logical communication, such as information that allows for theextraction of location information. In some examples, Transceivers 1913that include infrared transmitters have calibrated maximum intensities,and the logical communication levels received may be used to determineadditional confirming information related to location. Various smartdevices and/or position determining devices described herein may includeor be equipped with an infrared emission element that may serve as aTransceiver supported by an Agent 1913 and used to indicate position anddirection orientation.

In some examples, the aspects of FIG. 19A may represent a virtualviewing environment that a user, such as an architect or engineer may beimmersed in with a viewing apparatus, such as a Virtual Reality headset.The user may utilize localization and direction orientation aspects ofthe camera systems to drive the content of a virtual display view planeand a virtual location within a virtual model being displayed. In someexamples, a user may be located at a building site configured with acamera system such as illustrated in FIG. 19A while an architect may belocated in a separate room configured with a similar camera system asillustrated in FIG. 19A. In some of these examples, the architect mayobserve the view perspective of the user in the real-world location. Insome other examples, the architect may occupy a virtual location andobserve, through camera output of the real-world location, both thereal-world location, the user in the real-world location and a displayof the virtual model.

Referring to FIG. 19B, in some specific embodiments, ancillary positiondetermining devices may include an extension apparatus 1966 supported byan Agent. The extension apparatus 1966 may include, for example apointer 1960. The pointer 1966 may include a fixed length of rigid orsemi-rigid material, or a telescopic combination of multiple lengths ofrigid and/or semi-rigid materials. The pointer 1966 may be configuredwith areas of one or more wireless transceivers 1961-1963 at variousdistances from a first point 1963 of the pointer 1960. A location of thefirst point 1963 may essentially be the tip, or other convenient area.

A second area containing one or more transceivers 1961 and 1962 may beused as indicators that will be detected by directional apparatus, suchas an infrared camera. A user may direct a pointer 1960 in a directionof interest and engage an activation mechanism, such as a switch, orengage in a motion to indicate the time to obtain the position anddirection orientation. For example, an agent may activate a switch 1965to activate a Transceiver 1960-1963 and partake in logical communicationwith a Reference Point Transceiver 1967. In some embodiments, thelogical communication may be manifested as a pattern of light. Acontroller may be supplied with the pattern of light transmitted as wellas Reference Position information and generate a direction of interest.

According to the methods of the present invention, position points P1-P4may be generated based upon the logical communications between theReference Point Transceiver 1967 and the Transceivers supported by anAgent 1960-1964. A vector 1968 may be generated based upon the positionpoints P1-P4. In addition, a smart device 1964 may also communicate withthe Reference Point Transceiver 1967 and a position point P5 associatedwith the smart device 1964 may be generated.

In some embodiments, a vector 1969 may be generated based upon theposition point P5 of the smart device 1964 and a position point P1-P4generated based upon logical communications with Transceivers 1960-1963located on or within the ancillary position determining device 1966.

Referring now to FIG. 19C as discussed further herein, a sensor thatincludes a microelectromechanical system (MEMS) accelerometer may beused to track vibration patterns. In some embodiments, a MEMSaccelerometer 1905 may be included within a smart device, such as atablet or a smart phone 1904. Other embodiments include a sensorindependent of a smart device. Still other embodiments include a sensorpackaged with a controller for executing software specific to thesensor, such as the Fluke™ 3561 FC Vibration Sensor. A structuralcomponent 1901 of a structure for which conditions will be monitoredwith sensors may include a vibration integrator 1902 with an attachmentfixture 1903 that establishes vibrational integrity between anaccelerometer 1905 in a smart phone 1904 and the structural component1901. The vibration integrator 1902 may be matched via its shape andmaterial to accurately convey vibrations present in the structuralcomponent to the accelerometer 1905 in the smart device 1904. In someembodiments a vibration integrator may include a damper or filter toexclude certain frequencies that may be considered noise to someapplications. A damper may be directional such that only vibrationfrequencies in a particular direction are excluded.

It is understood that an accelerometer 1905 does not need to beincorporated into a smart phone and may be directly fixed to anattachment fixture 1903 or fixed to a vibration integrator 1902 or fixedto a structural component 1901.

Vibrations present in the structural component may be indicative of astate of functioning of equipment included in the structure (not shownin FIG. 19C). For example, a first pattern of vibrations, which mayinclude frequency and/or amplitude and variations of one or both offrequency and amplitude may indicate a proper functioning of a piece ofequipment. Patterns of equipment installed in a setting in a structuremay be recorded under proper operating conditions to set up an initialproper state of functioning. Patterns derived from a subsequent sensorreading, such as an accelerometer 1905 reading may indicate a variationfrom the initial pattern of sufficient magnitude to indicate amalfunction or wear present in the equipment.

In some embodiments, a user, such as a service technician, may installan accelerometer into the attachment fixture for the specific purpose oftaking an accelerometer reading. A smart phone 1904 may run an app thatrecords a time and place and vibration pattern received. The vibrationpattern may be compared with a known set of vibration patterns and acondition of the structured may be ascertained from the comparison. Thetime date and vibration pattern may be transmitted to a server andaggregated with other sensor readings.

In another aspect, in some embodiments, a second accelerometer 1905A maybe used to introduce a vibration pattern into the structural component1901. The second device may include a second attachment fixture 1903Athat establishes vibrational integrity between the second accelerometer1905A in a second smart phone 1904A and a second vibration integrator1902A. The vibration pattern introduced may include a known frequencyand amplitude. In some embodiments, the vibration pattern will include asequence of frequencies and amplitudes, wherein different frequenciesand amplitudes will be effective in diagnosing or otherwise indicatingan underlying causation for a pattern of vibration. The secondaccelerometer 1905A and the first accelerometer 1905 may be synchronizedvia executable software such that the first accelerometer will detectthe vibrations introduced by the second accelerometer 1905A. Anydiscrepancies between what was introduced by the first accelerometer1905A and the first accelerometer 1905 may be indicative of a state ofthe structure.

For example, introduction of a frequency pattern into a beam that issound may transmit well through the beam and be detected with minimalvariations from the frequency pattern that was introduced. However, abeam that is cracked or has rot within it may not convey the frequencypattern to the first accelerometer or convey the frequency pattern withsignificant distortion and/or diminutions in amplitude.

A history of sensor readings associated with a particular structureand/or group of structures may be stored and referenced to assist ininterpreting a cause for a particular vibration pattern.

Vibration sensors may be installed and recorded in as built data oradded to a structure in a retrofit. Some commercial sensors (such as theFluke 3561 FC Vibration Sensor) may be associated with vendor suppliedsoftware for ease of retrofit implementation.

According to the present invention, accelerometers or other vibrationsensors are deployed in specific locations and tracked in a structureaccording to the respective sensor location. In addition, a relativeposition of a particular sensor position is tracked relative to othersensors (vibration sensors or sensors for monitoring differentmodalities of ambient conditions). The present system includes an AVMthat may store and make available to a user and/or to AI applicationswhich structural components are in vibrational communication with aparticular sensor. Various sensors include underlying piezoelectric,accelerometers of other technologies.

Embodiments also include a sensor programmed to reside in a lower powerstates and to periodically “wake itself up” (enter a higher poweredstate) to take a reading and transmit the reading. Sensor readings maybe correlated with different types of wear, damage, failure or properoperation of components included in a structure. The AVM tracks locationand may rank a likelihood of a component responsible for a particularvibration pattern detected by a sensor. The ranking may be based uponproximity, mediums available for communicating the vibration pattern(such as a beam traversing a significant portion of a structure butwhich provides excellent mechanical communication for the vibration).

Some embodiments also associate a sensor reading of vibration with atype of motion likely to cause such a reading. For example, somereadings may include a linear component and a rotational component (suchas operation of a washing machine during certain cycles). Patterns ofnormal and abnormal operation may be recorded and deciphered viaprogrammable software on a controller.

In another aspect, a pattern of sensor data that denotes spikes oflinear data may be associated with a human being walking. Overtime, acontroller may track sensor reading patterns and associate a particularpattern with the walk of a particular person.

It is also within the scope of the invention to track and analyze a setof data associated with a primary signal and additional sets of data(secondary, tertiary etc.) tracking harmonics of the primary signal. TheAVM may also track sets of data associated with simultaneous, and/orclosely timed readings received from multiple sensors and associate anamplitude, sequence, delay or other attribute of the data sets relativeto each other to provide input as to a location of a source of thevibration. Additionally, vibration sensors may include axis within thesensor. For example, two axis and three axis sensors may have adirection of each axis included in the AVM and used in analysis of avibration pattern.

The present invention also provides simple and fast procedures for theprovision of directions of a User or a sensor to a source of vibrationbased upon analysis of readings of one or more sensors via the X. Y andZ location determination and directional ray or vector generationmethods described herein.

Disparate types of sensor may also provide disparate data types that areuseful in combination to determine a source of sensor readings. Forexample, a vibration sensor reading indicating erratic motion may becombined with an increased temperature reading from a sensor proximateto an item of equipment. The combined sensor readings may assist in ananalysis of a cause of the sensor readings.

In still another aspect, one or more sensor readings may be correlatedto a life expectancy of an item of equipment, such as for example aheating Ventilation and Air Conditioning (HVAC) unit. By way ofnon-limiting example, an ammeter sensor reading measuring an electricaldraw of an HVAC unit may be quantified upon deployment of the unit. Theinitial readings may act as a baseline of a unit in excellentoperational condition. A similar baseline reading may be taken via anaccelerometer measuring a vibration generated by the HVAC unit. Stillother sensor readings may include airflow, temperature, humidity, orother condition. Over time, a change in one or more sensor readingvalues may indicate some wear and move the HVAC equipment item into a“normal wear but operational” status.

Still further along a time continuum, one or more sensor readings mayindicate a pending failure. For example, a current required to run theunit may be measured by the ammeter sensor and indicate an increaseddraw in electrical current. Likewise, airflow may decrease, andtemperature may increase, and other sensors may provide additionalevidence of a pending failure. Finally, a failed unit may generate avery high temperature reading and ammeter readings may increase to alevel of sufficient electrical current draw to trip an electricalbreaker, thereby indicating a failure.

According to the present invention, any of the sensor readings (or all,or some subset of all sensor readings) may be referenced to generate analert. Following the alert, remedial action may be taken.

Referring now to FIG. 20, methods steps that may be executed in someembodiments of the present invention are presented. At step 2001, asdiscussed in detail herein, Transceivers may be affixed to referencepositions within or proximate to a structure. In some preferredembodiment's Transceivers are positioned at the perimeter of thestructure and are capable of wireless communication form any pointwithin the structure.

At step 2002, multiple sensors are deployed at positions within, orproximate to, the structure in a manner conducive for the sensor tooperate and generate data descriptive of a condition that is one or bothof: within the structure or proximate to the structure. The sensor willalso generate a digital signal descriptive of the condition monitored bythe sensor. Deployed may include affixing the sensor in a fashion thatenables to sensor in a manner intended. For example, an accelerometermay be fixedly attached to a beam or other structural component in orderto accurately experience vibrations emanating from the structuralcomponent. A temperature probe may need to be properly positioned to beexposed to ambient temperature conditions. An ammeter may be installedin a position enabling the ammeter to accurately determine an electricalcurrent being conducted by an electrical wire. Other sensors willlikewise be installed within the structure in a place and mannerconducive to generating accurate readings of conditions within thestructure.

At step 2003, positional coordinates may be generated for some or all ofthe sensors deployed. As discussed herein, positional coordinates may bebased upon triangulation between the sensor or a smart device proximateto the sensor in wireless communication with two, three or moretransceivers generating wireless transmissions from reference points. Aphysical position of the sensor may also be determined based uponwireless communication of the sensor with two or more of the wirelesstransceivers at the reference positions. As discussed herein thephysical position may include an X coordinate and a Y coordinate on anX, Y plane and an elevation based upon a Z coordinate relative to aground plane or other designated plane of origin.

At step 2004, the sensor is activated to an operational state andgenerates the digital data and transmits the data. In some preferredembodiments, the sensor will transmit the data via a wirelesstransmission. In other embodiments, the sensor may transmit the data viaan electrical or optical connector.

At step 2005, a first table is generated that includes conditions thatwould preferably precede deployment of the structure. For example, for aresidential structure, such as a single family home, a multi-familyhome, a condominium or a townhome; a first table may include buildingcodes. For other structures, ANSI codes, engineering specifications andother desired attributes may be included. Engineering specifications mayinclude, by way of non-limiting example, an amount of pressure (orweight) a load bearing wall or other structure may be exposed to. Otherspecifications may include an amount of vibration (such as vibrationcaused by wind or physical activity within the structure may be safelyhandled by the structure without causing damage and/or making thestructure unsafe.

At step 2006, the deployed sensors are operational to measure multipleconditions within the structure. The measurements are made, for example,via physical operation of the sensors, which may include applying anoperating electrical current to active components within the sensor.Operation may be based upon a schedule of periodic readings, via remotecontrol, or resulting from manual manipulation of a control on thesensor.

At step 2007, a time registration indicating when values of one or moreconditions were measured is recorded. In some embodiments, the timeregistrations may be utilized as an index to synchronize multiple sensorreadings. An index may be activated based upon a time at which a sensorwas activated. The index may be used to create a chronological sequenceof sensor readings. The index may also be used to synchronize multiplesensor readings and thereby capture a holistic picture of a structureduring a given time period.

At step 2008, a physical location of the sensor may be determined. Adirection of the sensor from a home position may also be determined viathe processes described herein. A distance of sensor to an item ofequipment or an area of interest may be determined. The distance may bedetermined via methods, such as LIDAR, echo, User entry or other method.

At step 2009 a digital signal is transmitted descriptive of thecondition of the structure. The condition of the structure may be basedupon Vital Conditions of the structure assessed via the sensor readings.The transmission may include a logical communication with wirelesscommunications via an industrial scientific and medical (ISM) bandwavelength which may include wavelengths between 6.765 MHz and 246 GHz.WiFi is one protocol that may be implemented for wireless communication,as is Bluetooth, ANT, infrared or other protocol. At step 2010, aphysical state of the building at the physical position of the sensor,or an area of the structure within range of the sensor, may becorrelated with the digital signal descriptive of a condition of thestructure. A condition present in the structure at an indexed time, maybe correlated with a physical state of the structure or a conditionpresent in the structure at the correlated time.

At step 2011, preferably the location is quantified via an X, Y, and Zcoordinate. The sensor locations and/or areas of the structure withinrange of the sensor for which the sensor may take a reading, areassociated with location coordinates, such as X, Y and Z coordinates.

At method step 2012, a controller may compare measured values ofconditions present in the structure with the conditions precedent todeployment of the structure. For example, the conditions precedent todeployment may include metrics of safety factors. Metrics of safetyfactors may include building codes, stress indicators, load capacity(weight), electrical current drawn, water pressure minimum and maximums,humidity, particulate levels in air, presence of mold or spore forms,presences of insects or rodents, etc.

Referring now to step 2013, a determination may be made as whetherconditions precedent to deployment of a structure are present within thestructure at a given point in time. In some embodiments, thisdetermination may be made based upon the measured values of sensorsindicating conditions within the structure and the time registration.

At method step 2014, a transmission may be made that indicates that astructure has not met a condition precedent to deployment. Thetransmission may include an alert that the structure is somehowdeficient and that a remedial action may be one or both of necessary andappropriate.

In some embodiments, a condition precedent may be indicative of a valueof a property for use as collateral to an obligation. For example, aproperty that includes a structure that is not up to code may not be asvaluable as a property that meets all codes and/or engineeringspecifications. In another non-limiting example, a value designated fora property that includes a structure may be monitored in real time, orat least in near real time (with no artificial delay introduced by anyaspects of the system and the only delay resulting from timingconsiderations entailed in normal operation of the system (e.g.transmitting times, processing times; storage times, communication speedand the like).

Referring now to FIG. 20A, additional steps that may be implemented insome embodiments are illustrated. Method steps may include utilizingaccelerometers and/or other sensors. At step 2015 a vibration isintroduced into a component of the structure and at step 2016 thevibration pattern introduced is compared with a pattern of vibrationdetected by a MEMS. At step 2017 a series of transitions of MEMSaccelerometer readings may be tracked and at step 2018 the pattern ofvibrations measured by the MEMS accelerometers may be correlated withstructural integrity.

At step 2019, alternatively, a condition of a structure may be generatedand based upon a pattern of vibration measured (transmitted and receivedwithin the structure). At step 2020, a threshold range of valuesmeasured by a sensor may be set and one or more method steps may beexecuted based upon whether the threshold range is exceeded or not.Method steps may also be predicated upon an analysis that indicates aparticular condition or set of conditions within the structure.

Referring now to FIGS. 20B and 20C, method steps are described for agentposition location. At step 2022 a transmission of a layer one locationdesignation may be performed of at least one agent supported positiontransceiver.

At step 2023 a receipt of a designation of a layer two positionreference transceiver based upon the layer one location may occur.

At step 2024, a calculation of a distance from multiple layer twoposition reference transceivers and at least one agent supportedtransceiver may occur.

At 2025 the operations of step 2024 may be repeated for multiplecalculated distances until a position of the agent supported transceivermay be calculated based upon these calculated distances.

At step 2026, again measurements may be repeated multiple times wherethe same frequency is utilized.

At step 2027, again the measurement steps may be repeated multipletimes; however, utilizing different frequencies.

At step 2028, the various data may be used to calculate an estimatedposition of the at least one agent supported position transceiver basedupon a mathematical process.

At step 2029 a vector may be generated based upon at least one of theposition of the agent supported transceiver based upon the calculateddistances and the estimated position of the at least one agent supportedposition transceiver.

Referring now to FIG. 20C, method steps for a process that may bepracticed using a same or different transceiving modality isillustrated. At method step 2030 a location of a first PositionTransceiver supported by an Agent calculated based upon wirelesscommunication with a first Reference Position Transceiver. The wirelesscommunication may include transmission and receipt of logicalcommunications via one or more of the wireless modalities discussedherein, and therefore may include, but is not limited to modalitiesincluding: WiFi communications, UWB communications, Ultrasoniccommunications, Infrared communications, Bluetooth communication, andother wavelength communications.

Generally, triangulation based upon wireless communication determinesthree distances from a first point to a second point based upon wirelesscommunications using a single modality of communication. According tothe present invention, triangulation may include a single modality, ormultiple modalities of wireless communication. Also, within a singlemodality, a same or multiple wavelengths may be used for communication.For example, a first distance may be determined based a wirelesscommunication in a WiFi modality using a first bandwidth. A seconddistance may be determined based upon a wireless communication in thesame WiFi modality and using the same first specific bandwidth, oranother bandwidth within the same WiFi modality. Alternatively, or incombination with the determination of the second distance based upon theWiFi modality, the second distance may be determined based upon awireless communication in a Bluetooth (or other communication modality,such as those discussed herein). A third distance may also be generatedand may be based upon a wireless communication in any of a WiFimodality, a Bluetooth modality, or other communication modality (such asUWB, infrared or ultrasonic modalities).

According to the present invention, almost any combination ofcommunication modalities may be used for wireless communications uponwhich a distance is generated. A controller may combine each respectivedistance to generate a location of a Transceiver involved. Themulti-modality communication Transceivers taught in the presentinvention enable a wide range of combinations that may be leveraged toprovide beneficial aspects of a particular modality with otherbeneficial aspects of another communication modality.

For example, an ultrasonic or infrared communication modality mayprovide a very accurate distance determination. In addition, theultrasonic and infrared distance determinations may include adirectional aspect that may be leveraged to provide a more accurateposition determination when combine with an omni-direction communicationmodality based distance, such as a distance based upon WiFi and/orBluetooth. In addition, ultrasonic and/or infrared communicationmodality may provide an accurate determination of a room in which themeasurement is taking place since these communication modalities may notpass through solid objects (such as a wall) very well. Other modalitiesmay each provide advantages and/or disadvantages based upon variablesthat may have an impact upon a wireless communication.

The present invention therefore includes using multiple communicationmodalities and multiple bandwidths within a single communicationmodality to determine a particular distance (e.g. a distance betweenReference Point Transceiver 1 to a Transceiver supported by an Agent)and also using multiple communication modalities and multiple bandwidthswithin a single communication modality to determine different distances(e.g. a first distance between Reference Point Transceiver 1 to aTransceiver supported by an Agent and a second distance betweenReference Point Transceiver 2 and the Transceiver supported by theAgent).

In some embodiments, a choice of communication modality may be basedupon environmental conditions in which the wireless communication istaking place. For example, an environment with significant particulatein the air (such as smoke or dust) may not be ideal for an infraredwireless transmission, or an environment with significant electricalnoise may be detrimental to a WiFi or Bluetooth wireless communication,but not detrimental at all to an ultrasonic (or infrared) wirelesscommunication. Therefore, the present in invention includes the abilityto receive data from sensors proximate to an area of wirelesscommunication (as described in this disclosure) and to weight aconsideration of a particular wireless communication based upon datagenerated by such sensors.

At method step 2031, a location of a second position transceiver alsosupported by the Agent is generated and based upon a wirelesscommunication with a Reference Position Transceiver. The ability toutilize multiple modalities and multiple frequencies described abovealso applies to this position generation (and other positiongeneration). The second position Transceiver may also be included in asame device as the first position transceiver, such as a smart phone(and in some embodiments may include the same position Transceiver at adifferent location) or the second position Transceiver may beincorporated into a separate device, such as a ring, a watch, a pointera wand or other device.

At method step 2032, a vector is generated based upon the location ofthe first position Transceiver and the position of the Second positionTransceiver. The vector will generally include a position of the Agentbased upon the wireless communications and a direction of interest basedupon a relative position of the location of the first positionTransceiver and the location of the second position Transceiver.

At step 2033, a query of a database may be made based upon the vectorgenerated. The database query may include, for example, a request fordirection to a defined area of interest or a defined object or aspect ofthe structure; a request for a portion of an AVM, details of an aspectof a structure (such as electrical features, plumbing features orstructural features) or almost any other database information that maybe related to a position of the Agent and a direction of interest.

At step 2034, a user interface may be generated based upon a response tothe query based upon the vector. The user interface may include one orboth of aspects ascertainable to a human Agent (human readable and/orpictorial) and aspects ascertainable to an automation Agent (such asdigital data).

At method step 2035, the user interface may be displayed upon a screenincluded in a smart device associated with the Agent.

Referring now to FIG. 21, an exemplary method for hierarchical actionsbased upon structural conditions is shown. At 2101, the aggregationserver receives an electronic communication comprising an obligation anda remediation table. An obligation could be based off a contract(traditional contract or smart contract) and reflect any obligationsurrounding the property, such as one or more of: maintain a certainloan-to-value ratio; maintain habitable living conditions by referenceto pre-defined Vital Conditions; maintain a certain occupancy level; orachieve any Objective. One or more sensors are associated with eachobligation. For example, if the obligation is to maintain habitableliving conditions, associated sensors may include one or more of:thermometer, humidity meter, particulate concentration detector (e.g.,smoke detector), or a vibration detector. Additionally, the remediationtable may include one or more ranges for each sensor. For example, insome embodiments, if the obligation is to maintain habitable livingconditions, then the acceptable range of temperatures as measured fromthe thermometer sensor might be 65-80° F. The remediation table alsoincludes one or more prescribed remedial actions to take if the sensorreading falls outside an acceptable range. For example, if thetemperature of the structure reaches 85° F., then the remedial actionmay be one or more of: increase the power of the air conditioning unit,shut a door, or shut a window. There may be one or more sets ofacceptable ranges, each with its own respective remedial action. Forexample, if the acceptable temperature range is 65-80° F., a secondtemperature range may be 50-90° F. Within this temperature range, thestructure may still be able to comply with the obligation if specificsteps are taken. Here, closing a door might not restore the temperatureto the acceptable range, but replacing the roof might be sufficient. Anadditional possible sensor range might reflect that the structure isincapable of ever complying with its obligations. Continuing with thethermometer example, a third temperature range may be 0-1100° F. (thelatter being the average temperature of a house fire). In this case, thestructure may be deemed permanently non-compliant with a criterionassociated with the structure.

A contractually imposed obligation may include, for example, that agiven Structure always be at a C3 level or higher. A Structure's UADCondition Rating is, in part, a function of the frequency and necessityof repair of major components. A Structure reaches C4 when it hasimprovements that feature some “minor deferred maintenance” and requires“only minimal repairs to components/mechanical systems.” Id.Accordingly, the AVM, supplemented with SVCM, is a desirable way toensure a Structure does not reach C4, in at least two ways. First, insome embodiments, the AVM tracks maintenance history of variouscomponents. In some embodiments, the AVM may be pre-loaded with adesired repair schedule. For example, experts recommend that the HVACsystem be serviced at least once per year. Each of these services may bedocumented in the AVM through the successful completion of a ServiceCall. Conversely, if the AVM does not have a record of maintenance ofthe HVAC system for three years (as an example), then this lack ofmaintenance may constitute deferred maintenance sufficient to change therating of the Structure from C3 to C4.

Second, Sensors may be placed to monitor components, mechanical systems,and electrical systems in accordance with SVCM. For example, Sensors mayinclude voltmeters positioned throughout the Structure's electricalsystems. If a voltmeter detects an unexpected, prolonged voltage dropacross the voltmeter, the Structure may also be downgraded to C4.

Other factors affecting the appraisal value of a Structure or its UADCondition Rating that may be reflected in the AVM include, withoutlimitation: heat and air throughout the Structure, Structuralconstruction materials and updates, neighborhood value (where otherStructures in a neighborhood are also covered by an AVM and SVCM),structural soundness and sealing, and appliance maintenance.

objective list to put property in rated condition (A ratedproperty—specific contractual commitment—property will be kept in thisrated condition—do sensor readings reflect that they meet—if you have 20items, out of compliance with 7, it can do that—evaluate price ofremedial action—hierarchically rate remedial actions—combining logicw/physical measurements, measuring states, flagging, giving indicationof what to do today—can look at portfolio today

At step 2102, multiple Sensor readings are aggregated at the aggregationserver over time. These may be multiple readings from one Sensor, asingle reading from multiple Sensors, or multiple readings from multipleSensors. This may serve to establish a pattern of measurementspertaining to the objective. Additionally, in some embodiments, anentity may have access to aggregation data for a plurality ofstructures. Such data can be additionally combined to usefully evaluateparameters and desired ranges. For example, upon the purchase of astructure, an appraisal or a construction of an AVM may indicate thatthe optimal temperature for the structure is 74° F. However, if 99 othersimilar structures maintain an average temperature of 71° F., then theacceptable range may be modified accordingly.

At step 2103, periodically the aggregation server may determine whethera measurement taken from a Sensor falls within the acceptable range.Periodically may mean daily, weekly, monthly, yearly, or any otherinterval. As discussed at step 2101, multiple Sensor ranges may beconsidered. At steps 2104 and 2105, these determinations may be sent toa human-ascertainable interface. At step 2104, a determination may berequested from the human-ascertainable interface, and if the sensorreadings fall within an acceptable range, the aggregation server mayprovide confirmation of same.

At step 2105, however, if the determination is that the measurements donot fall within an acceptable range, the remediation table is consultedto determine the appropriate remediation action. As discussed at step2101, the remediation action may comprise changing the conditions of thestructure. In some embodiments, the AVM may be consulted to determinethe most effective remediation measure. In some embodiments, theeffectiveness of a remediation measure is determined with reference toone or more of: the speed of the remediation, the cost of theremediation, or the long-term effect of the remediation.

At step 2106, in some embodiments, the aggregation server is in logicalconnection with the AVM, home automation tools, or other automatedapparatuses that can implement the associated remedial action. Forexample, if the temperature measurement falls outside the acceptablerange, in these embodiments the aggregation server can communicate withan automated thermostat to adjust the temperature accordingly (e.g., byincreasing power to the air condition or heater). In some embodiments,the remediation table comprises a hierarchical listing of actions totake to fall into compliance with an obligation. For example, thecontract or other communication specifying the obligation may enumerate20 conditions to monitor, such as yearly HVAC maintenance, airflowthroughout the Structure, and insulation quality in each room of theStructure. If the AVM indicates that a predetermined threshold number ofthese conditions are not met, then the aggregation server may takeremedial actions in accordance with a set of predefined rules. Theserules may be prioritized by, for example, price or efficacy inremediating the undesired condition. Based on the previous example, theaggregation server may determine that the HVAC has not been servicedwithin the last three years; a crack in the wall exists in the livingroom; and the kitchen is absorbing outside heat at a greater rate thananticipated upon appraisal or at the time of the obligation. If theremedial action rules are prioritized by price, then the aggregationserver may determine that the least expensive remedial action isrequesting service to the HVAC—and the aggregation server mayautomatically order such service. At step 2107, a remedial action may becommenced via an automation.

Referring now to FIG. 22A, method steps that may be implemented in someembodiments of the present invention are illustrated. At method step2200, a service request is originated. The service request may beassociated with geospatial location services are used to determinegeospatial location such as a location of the structure with a positionand direction of interest. At step 2201 the geospatial services may beused to determine an Agent's location relative to the structure anddirections thereto. The methods used may include, by way of non-limitingexample, one or more of: satellite-based GPS, cell tower triangulation,radio signal triangulation, Wi-Fi signal location services, infraredtransmitters and the like.

In some embodiments, geospatial location services may becross-referenced with a database registry of as built virtually modeledfacilities and may be used in conjunction with a network of registeredservice technicians to route a nearest available service technician to astructure related to the service request.

At method step 2202, the Agent's entry into the structure will beregistered. Registration of entry into the structure may be achievedthrough multiple methods, which may include, by way of non-limitingexample, on or more of: WiFi gateway detection, infrared detection,magnetic door locking systems, Bluetooth services, and the like. Uponentry into the structure requesting the service call, system willregister the service technician's entry into the structure.

At method step 2203, a series of accelerometer readings may generatetravel log of Agent travel within the structure and may be based upon achange in triangulation values and/or an accelerometer reading or imagebased movement techniques. The methods used may be, by means ofnon-limiting example, one or more of: use of data gleaned fromaccelerometers located on or in possession of Agent, WiFi services,radio frequency (RF) triangulation, Bluetooth technology, infraredtransceiving, ultrasonic transceiving, RFID badges, and the like.

At method step 2204, a smart device associated with the Agent will beregistered for communication with the Agent.

At method step 2205, a unique identifier may be associated with thesmart device and/or the Agent.

At method step 2206, a purpose of the Agent presence within thestructure is recorded. Exemplary purposes may include, one or more of: aservice call placed from the structure to the system detailing anequipment malfunction; service calls from the structure detailingnon-specific malfunctions and symptomatic data indicating an equipmentmalfunction; a service call placed by self-assessing equipment utilizinginternet of things (IoT) and machine learning functionality to ascertainmalfunctions and predictive analytics to anticipate malfunctions, andthe like. The system may integrate data reports into the AVM and relaycontent of the reports to the smart device associated with the Agent inthe field.

Alternatively, a smart device may arrive at a structure without priorknowledge of a purpose. Upon entry into the structure and registrationof the smart device a controller may relay data gleaned from the AVM,operational data uploaded to the system through IoT processes, and otherexperiential data reported to the controller. Methods for relation ofsuch data to an on-site smart device may include, by means ofnon-limiting example, referential data based on proprietary orienteeringprocesses to determine smart device location within structure, whichlocation will be cross-referenced with AVM data.

At method step 2207, a purpose for the presence of the Agent in thestructure may be recorded, such as for example in a log linked to theAVM. The method steps may use one or more of: augmented realityoverlays; heads-up displays or other wearable technologies; augmentedreality overlays displayed on smart devices; direct instructionalvectoring provided to the smart device by the system over WiFi internetconnection or LTE signal; virtual reality walkthrough instructionsprovided to a user interface and/or the smart device on site associatedwith the Agent, updated map/schematic displays detailing the structureand directing the Agent to a source of a purpose for being within thestructure based upon vectoring and orienteering processes. Orienteeringapparatus may include, by means of non-limiting example, on or more of:WiFi triangulation, infrared position detection, radio frequency (RF)detection, RF ID tracking, onboard accelerometers located on the smartdevice or carried smart devices, and the like.

Referring now to FIG. 22B, at method step 2208, a position Transceiverlocation within the structure along X and Y coordinates may bedetermined. At method step 2209, the position Transceiver's locationalong the Z axis may be determined. In some examples, the method steps2208 and 2209 may occur simultaneously or nearly simultaneously. Themethods used may be, by means of non-limiting example, one or more of:onboard magnetometers, onboard barometers, onboard accelerometers, andthe like, used in conjunction with in-structure XY axis positionprocesses described in method step above, along with data detailed inthe AVM of the structure.

At method step 2210, an Agent may designate and area of interest.

The system may determine an Agent's direction of interest using, bymeans of non-limiting example, on or more of the following methods:infrared pointers, laser direction finding devices, onboard camera(s),RF ID trackers, RFD finders, barcode scanners, hex/hash code scanners,WiFi triangulation, and the like.

At method step 2211, the Agent's distance and a pathway to an area ofinterest is generated and displayed upon a human readable interface. Themethods used may be, by means of non-limiting example, one or more ofthe following: infrared pointers, laser pointing devices, WiFitriangulation, RF ID sensors, RFD, depth perception sensors containedwithin onboard cameras, onboard magnetometers, Bluetooth technology, ANTsensors, directionally enabled smart device cases, and the like.

At method step 2212, records of equipment and/or area of interest willbe accessed and relayed to the smart device. The smart device'sposition, direction of interest, and distance to the equipment/area ofinterest may be cross-referenced with the AVM and experiential data maybe accessed and displayed on the user interface. Data regarding thepurpose may be added to the AVM and experiential data displayed to auser interface, such as a user interface on a smart device.

At method step 2213, symptomatic data will be diagnosed to determinecause of emergency as possible. The methods used may be, by means ofnon-limiting example, one or more of: IoT experiential data gathered andcollated from multiple sources across multiple facilities similar to thepresented symptomatic data, internet-gathered data analyzed by variousmachine learning technologies, algorithmic analytics of symptomatic datato determine causal indications, and smart device expertise.

At method step 2214, information related to the purpose and a subjectlocation is recorded and added as experiential data to the AVM. Themethods used may be, by means of non-limiting example, on or more of:equipment self-reporting through IoT and machine learning technologies,smart device entered data, experiential data gathered from emplacedsensors and other recording devices within the structure itself, and thelike.

Referring now to FIGS. 23A-23C, aspects of AR Headgear 2314 according tosome embodiments of the present invention are shown. In FIG. 23A, ahelmet type mount 2314 may be secured or otherwise fitted onto a user2301. The helmet 2314 includes one or more wireless position devices2310-2313 fixedly attached to the helmet 2314 or incorporated into thehelmet 2314. The wireless position devices 2310-2313 may be madefunctional in the same manner as wireless position devices 1303-1310illustrated in FIG. 13 and also interact with Reference Transceivers1311-1314 in a manner congruent with the discussion of ReferenceTransceivers 1311-1314. Although four wireless position devices2310-2313 are illustrated, any suitable number may be included, such asby way of non-limiting example a wireless position device on each sideof a helmet, a wireless position device 2310-2313 on a front portion ofthe helmet 2314 and a wireless position device 2310-2313 on a rearportion of the helmet, and a wireless position device 2310-2313 placedgenerally about 180 degrees across from each other. Embodiments mayinclude configurations of wireless position devices 2310-2313 in almostany pattern conducive to transmission and reception, accordingly, acirclet of wireless position devices 2310-2313 may be formed into acrown of contiguous wireless position devices 2310-2313, or a mesh ofwireless position devices 2310-2313 that covers all or some portion ofthe helmet are within the scope of the present invention.

In another aspect, each of the wireless position devices 2310-2313 mayinclude an array of wireless transceiver devices. In this manner, theredundancy of multiple transceivers may allow for more reliable and moreaccurate communication. Transceivers that operate at less than optimumperformance, or less than a median of performance may be ignored so thatthe logical communications remain clean through the transceiversoperating at a higher level.

A distance 2320 or 2321 between two wireless position devices 2310-2313will be sufficient for some modalities of wireless communication (e.g.ultra-wideband, IR, WiFi, GPS, etc.) to allow for respective positioncalculates to indicate w forward direction 2322 for the helmet, andaccordingly, for the user wearing the headgear 2314. Other directionsmay be determined in a similar fashion.

A controller 2319A may be attached to, or resident within the head gear2314 essentially enabling the headgear 2314 to function as a smartdevice or an auxiliary smart device as discussed throughout thisspecification. A power source 2319B, such as a battery or other storageof potential energy may also be included. The power source may be usedto power one or more of the controller 2319A; wireless position devices2310-2313; a display 2316 or other device included in the headgear 2314.

One or more visual display devices 2316 may be supported via a displaysupport 2315 in a position in front of the user's eye 2317 or otherwisewithin a field of view 2318 of the user 2301.

One or more image capture devices 2323 may also be fixedly attached tothe helmet 2314. In some embodiments, the image capture devices 2323 maybe used to capture As Built image data and/or general conditions of astructure. In another aspect, multiple image capture devices 2323 may beused as stereoscopic cameras to determine a distance of the helmet froman object within the field of view of the cameras 2323. Similarly, alaser device 2324, such as a LIDAR device, an infrared device, or otherdevice type may be fixed to or incorporated into the helmet 2314. Thelaser device 2324 may be functional to determine a distance of thedevice to a first reflective surface, and/or to scan a surface in frontof the helmet 2314. Laser devices pointing in a direction other thanforward may also be used with the same functionality in a differentdirection.

FIGS. 23B and 23C illustrate headgear 2331 and 2340 of types other thana helmet. In FIG. 23B a hat type headgear 2331 may include wirelessposition devices 2330-2333 at various position on a hat 2331, such as acrown of a hat 2332; a rear portion of a hat 2330 and a brim of a hat2335. In addition, a display fixture 2333 may support a user viewabledisplay 2334 that is within a field of view 2337 of an eye 2336 of auser 2301.

FIG. 23C illustrates a pair of eye glasses with additional functionalityand components to qualify it as AV Headgear. The eyeglass AV Headgear2342 may include wireless positional devices 2340-2341 that are fixed toor incorporated into the eyeglass AV Headgear 2342. The wirelesspositional devices 2340-2341 may be placed in any position convenient,such as, for example, a WPD 2340 on the temple of the eyeglass AVHeadgear 2342 or a WPD 2341 on the bridge of the eyeglass AV Headgear2343 in proximity to an eye 2344.

Referring to FIG. 24A, a sample interior map 2401 is shown. The map 2401comprises an indicator of a first position of the emergency responder2402, an entrance 2403, a recommended path 2404, and a second position2405. The second position 2405 may include an area requiring theemergency response, a problem area to investigate, the location of anobject or person to be rescued, or stairs leading to same. The emergencyresponder's first position 2402 is related to the second position andone or more vectors through the physical Structure from the firstposition to the second position are established. In some iterations ofthe present invention, the vector may be established by relatingcharacteristics of the emergency responder to the suitability of theemergency, and the conditions present in the Structure. Conditionswithin the Structure are monitored by a Structure-wide building vitalstatistics monitoring apparatus which may produce readings of, by way ofnon-limiting example, one or more of: carbon monoxide monitoring; IRsensor monitoring; air quality monitoring; and the like. Accordingly, anappropriate vector for an emergency responder may be established. Theestablished vector between the first position and the second position isrelayed to the emergency responder via a user interface and theemergency responder's progress along the vector is monitored accordingto monitoring protocols.

Referring to FIG. 24B, a depiction of Augmented Reality HUD 2410application of the present invention is displayed. An emergencyresponder may access the AVM data of the subject Structure and theorienteering function of the AVM is engaged. The orienteering functionof the AVM relays a pathway 2411 via waypoint 2412 from the emergencyresponder's first position to a second position. The orienteeringfunction directs the emergency responder on a pathway 2411 through asubject Structure around As Built emplaced elements 2413 oremergency-related obstacles 2414, which may impede a more direct routeto a waypoint 2412 during an emergency.

In preferred embodiments of the present invention, wearable augmentedreality HUD technologies 2410 are utilized. The pathway 2411 to anemergency responder's waypoint 2412 while responding to an emergency isdisplayed via augmented reality overlay through the subject Structuremaking use of As Built AVM data to direct an emergency responder aroundemplaced objects 2413 or emergency obstacles 2414 otherwise obstructinga given user's pathway 2411 to a waypoint 2412.

A user may be located within a structure and may wear an augmentedreality headgear. The user may establish an orientation for his viewplane in numerous manners and then view information, imagery stored dataand the like related to that view plane.

Referring now to FIG. 25A, an illustration of a user 2500 employing anaugmented reality headgear 2510 in concert with a designation ofdirection 2503 is provided. The means of establishing a direction andorienting spatially may be used. For example, mobile device 2501 may bemoved in the direction of the view plane which will correspond to theoriented direction 2503. The mobile device may include GPS and/orcellular-based location schemes to extrapolate the direction fromlocation information. In some examples, the user 2500 may be requestedto orient the system by displaying a message 2511 to the user via theaugmented reality headgear 2510.

Once the user has oriented his system, he may interact with databaseservers to access and create database records of various types includingimagery. Referring to FIG. 25B, an illustration of a user 2500 employingan oriented Headgear 2510 in a use mode is provided. In some examples,the orientation may be conducted with a smart device 2501. In theaugmented reality headgear various types of data may be presentedinclude a textual interface 2520 of command options and information.Additionally, a demonstration of imagery being collected by smart device2501 of the environment may be displayed as image 2521.

Referring to FIG. 25C, aspects of information display are illustrated.The figure illustrates a user 2500 wearing an oriented augmented realityheadgear 2510 and holding a smart device 2501. In an example, the figureillustrates displaying stored information about theStructure/environment in which the user 2500 is oriented. In an example,an image 2530 displaying a wall structure and embedded facilities andother such structures is provided. The display may also include othertypes of data presented such as textual information 2531.

In some examples, an augmented reality headgear may also include camerasystems of various kinds to record the view plane of the user. In someexamples, a stereoscopic camera system may be incorporated into theaugmented reality headgear. Referring to FIG. 25D, a Headgear equippedwith location viewing stereoscopic cameras 2515 is utilized by a user2500 in an interactive use mode. In the example, the user 2500 mayestablish orientation by moving in an orienteering direction. In someexamples, the augmented reality headgear may include GPS or cellularcommunication devices which may be able to sense the direction ofmovement and orient the system. In the augmented reality headgear,textual instructions 2535 may be provided to the user 2500. An inset2536 may provide a view of the current imagery observed by thestereoscopic cameras 2515.

Referring to FIG. 25E, an illustration of a user 2500 utilizing anoriented stereoscopic camera system 2515 to observe the oriented userview plane is shown. In an example, the augmented reality headgear maydisplay a stereoscopic rendering of the imagery 2541 utilizing suchfunction of the augmented reality headgear. In the illustration, aninset 2542 may include a display of historical data of the view planesuch as embedded structure and the like.

FIG. 25F illustrates a Headgear equipped with location viewingstereoscopic cameras 2515 in an interactive use mode to establishorientation by pointing in an orienteering direction while wearing aGPS-equipped device 2540. In some examples, a wearable GPS device 2540may be a smart watch. In other examples, any device that can provide alocation reference may be utilized. The device 2540 may use near fieldcommunication or other wireless protocols to communicate with receptionelements that may be located in the stereoscopic camera system totriangulate a location relative to the Headgear to provide orientationinformation. As in previous examples, the augmented reality headgear mayprovide textual information 2545 to the user 2500 and may providesimultaneous display of other information such as the camera view in aninset 2546.

FIG. 25G illustrates a Headgear equipped with location viewingstereoscopic cameras 2515 in an operational mode displaying historicinformation with current view inset. In some examples, a device 2540capable of providing location information may be worn by the user 2500.The virtual reality display may show a superposition of the view of thecamera system 2550 with previous information 2551 and historic views ofembedded structure, facilities and the like. Other information may bedisplayed in split screen or inset display 2552. In some examples, theuser may utilize the location sensing aspects of device 2540 to allowthe user to point to a virtual location displayed in his augmentedreality headgear. In some examples, sensors, haptic feedback elementsand the like may be additionally worn by a user such as in glove sensor2553 which may provide feedback to and from the user. The glove maysense aspects such as flexing motion that may occur, such as in agrabbing motion as a non-limiting example, such as movements of fingersand the like. As well, the user 2500 may feel physical feedback from thehaptic feedback elements to convey various aspects of the model in thevirtual reality display.

An important aspect of maintaining models of Structures andenvironments, as has been disclosed herein, may be to provide routineupdates to imagery and three-dimensional scanning of the location. InFIG. 25H, an example illustrates an oriented Headgear equipped withlocation viewing stereoscopic cameras 2515 used by a user 2500 inacquiring current picture data 2560 of a viewing direction. In someexamples, the augmented reality headgear may also include athree-dimensional scanning system 2562 that may record a three-dimensionmodel of the surfaces in the view plane. An inset 2561 may presentadditional data or views. In some examples, the user may utilize thelocation sensing aspects of device 2540 to allow the user to point to avirtual location displayed in his augmented reality headgear. In someexamples, sensors, haptic feedback elements and the like may beadditionally worn by a user such as in glove sensor 2553 which mayprovide feedback to and from the user. The glove may sense aspects suchas flexing motion that may occur, such as in a grabbing motion as anon-limiting example, such as movements of fingers and the like. Aswell, the user 2500 may feel physical feedback from the haptic feedbackelements to convey various aspects of the model in the virtual realitydisplay.

In some examples, the updating of location imagery may be supported bydisplaying information in the augmented reality headgear with anindicator of the updated information in a panoramic depiction. In someexamples, shading, coloring, or blinking may be used to depict imagerythat still has to be updated. Referring to FIG. 25I, an illustration ofa Headgear equipped with location viewing stereoscopic cameras 2515 isutilized by a user 2500 in an interactive use mode to record panoramicpicture data 2570 to update the status record. As the user recordscurrent imagery, the display may show the new unshaded imagery 2571 asdistinct from previously recorded regions of the environment 2572 thatneed to have new imagery recorded. In some examples, the user mayutilize the location sensing aspects of device 2540 to allow the user topoint to a virtual location displayed in his augmented reality headgear.As well, the user 2500 may feel physical feedback from the hapticfeedback elements to convey various aspects of the model in the virtualreality display.

There may be numerous manners to record new imagery andthree-dimensional structural data such as cameras, image detectors insmart devices, three-dimensional scanning devices and the line. FIG. 25Jshows an illustration of a user 2500 wearing a virtual reality display2510, and is illustrated updating current status capture images 2581 inpanoramic picture data 2580 with a smart device 2501 handheld camera inan interactive use mode to record panoramic picture data.

Referring now to FIG. 26A, method steps for Guided Orienteeringconducted by an Agent at a structure based on wireless communicationwherein a purpose of a Guided Orienteering is known are illustrated.

At method step 2600, a Guided Orienteering within a given Structure witha corresponding AVM is originated. Origination of a Guided Orienteeringmay be achieved by a variety of means including, by way of non-limitingexample, one or more of: automated notification of changes in thephysical state of a structure as recorded by a Structure VitalConditions Monitoring system, as disclosed in U.S. patent applicationSer. No. 16/165,517; smart contract and blockchain features designed toinitiate Guided Orienteering upon the achievement of pre-registeredlogical make/break points; audio communication, text message; emailcommunication; and the like.

At method step 2601, a geospatial location of an Agent is determined.This may be done by any of the well-known methods of coarse geo spatiallocation, such as triangulation involving GPS, Bluetooth, WiFi, UWB,ultrasonic and/or cellular communications.

At method step 2602, upon crossing a threshold into the Structure orproperty housing the Structure, an Agent's arrival at the subjectstructure or property housing the structure is registered with the AVMof the structure or property housing the structure. An Agent's arrivalat a subject structure may be achieved by one or more of, by way ofnon-limiting example, one or more of: geopositioning systems; in-groundradio frequency monitors; magnetic locking doors triggering make/breaksignals; IR sensors; camera recognition; RFID/Barcode/UUID scanning;optical monitoring or physical interaction with the subject structure.

At method step 2603, a smart device supported by an Agent comprises anaccelerometer, which assists in tracking movement of the Agentthroughout the building. The accelerometer can assist in trackingmovement by, for example, supplying acceleration data which can beconverted into position data using known methods of numericalintegration. In this way, the accelerometer can serve as a backuplocation finder to the WiFi triangulation discussed in considerabledetail above.

At method step 2604, the smart device supported by an Agent isregistered with the AVM of the Structure. Smart devices supported byAgents and Agents themselves may be selected for registration accordingto a number of factors including, by means of non-limiting example, oneor more of: relative proximity to a given Guided Orienteering's point oforigin; expertise or suitability relative to a purpose of a given GuidedOrienteering session; availability to partake in the GuidedOrienteering; cost of deployment; and the like. At step 2605, the smartdevice is associated with a specific Agent and a unique identifierassociated with the smart device is received.

At step 2606A-B, the Agent's purpose during the Guided Orienteering maybe registered with the AVM of the subject structure. Registration of theAgent's arrival at the subject structure may automatically cause theAgent's purpose on the Guided Orienteering to be registered by thestructure's AVM.

In some embodiments, at step 2606A, a purpose of the Agent ispre-registered in a system. Upon verification that the Agent has apurpose, at step 2607A, an interior map and instruction may be providedwith reference to the AVM. In some embodiments, a user interface isgenerated for the Agent on the Guided Orienteering.

By way of non-limiting example, a user interface may be one or more of:a smart device application, a virtual reality headset, an augmentedreality apparatus, a remote control interface for an unmanned vehicle,etc. In some embodiments, a user interface will relay information fromthe AVM relative to the Agent's position within the structure on a givenGuided Orienteering to the Agent in real or near real time. In someembodiments, a virtual representation of a physical apparatus or area ofinterest within the subject structure corresponding to the origin of theGuided Orienteering as stored in the AVM may be designated the secondposition and relayed to the Agent via the user interface.

Referring to FIG. 27A, a sample interior map 2701 is shown. The map 2701comprises an indicator of a first position of the Agent 2702, anentrance 2703, a recommended path 2704, and a second position 2705. Thesecond position 2705 may include a specific apparatus to be worked on, aproblem area to investigate, or stairs leading to same. The Agent'sfirst position 2702 is related to the second position and one or morevectors through the physical structure from the first position to thesecond position is established. In some iterations of the presentinvention, the vector may be established by relating characteristics ofthe Agent to the suitability of the Guided Orienteering, and theconditions present in the structure. Conditions within the structure aremonitored by a structure-wide building vital statistics monitoringapparatus which may produce readings of, by way of non-limiting example,one or more of: Carbon Monoxide monitoring; IR sensor monitoring; airquality monitoring; and the like. Accordingly, an appropriate vector fora given Agent may be established. The established vector between thefirst position and the second position is relayed to the Agent via auser interface and the Agent's progress along the vector is monitoredaccording to monitoring protocols.

Referring now to FIG. 27B, a depiction of Augmented Reality HUD 2710application of the present invention is displayed. An Agent on a GuidedOrienteering accesses the AVM data of the subject structure and theorienteering function of the AVM is engaged. The orienteering functionof the AVM on the Guided Orienteering relays a pathway 2711 via waypoint2712 from the Agent's first position to a second position. Theorienteering function directs the Agent on a pathway 2711 through asubject structure around as built emplaced elements 2713 which mayimpede a more direct route to a waypoint 2712 along a GuidedOrienteering.

In preferred embodiment of the present invention, wearable augmentedreality HUD technologies 2710 are utilized. The pathway 2711 to anAgent's waypoint 2712 in progress of a Guided Orienteering is displayedvia augmented reality overlay through the subject structure making useof as built AVM data to direct a given user around emplaced objects 2713otherwise obstructing a given user's pathway 2711 to a waypoint 2712 ona Guided Orienteering.

Referring again to FIG. 26A, in other embodiments, at step 2606B, theremay be no registered purpose for the Agent. In that case, at step 2607B,the Agent's position may be tracked on an interior map of the Structurewith reference to the AVM. This may allow, for example, securityofficers associated with the Structure to track a potential unidentifiedperson or may allow a person associated with a Structure to conveyappropriate instructions to the Agent.

By way of non-limiting example, a given Agent's position may beascertained and recorded by one or more of: relating the Agent'sposition to that of three or more wireless Transceivers affixed withinthe structure at known reference positions corresponding to virtualpositions within the AVM; IR sensor readings; GPS; cell signaltriangulation; trilateration and multi-lateration using emplacedsensors, which may use one or more of WiFi protocol, Bluetooth, etc.;accelerometers and/or magnetometers onboard a smart device or the Agent;optical sensors and the like.

Referring now to FIG. 26B, additional method steps for a GuidedOrienteering conducted by an Agent at a structure based on orienteeringwherein the purpose of a Guided Orienteering is known are illustrated.At steps 2608 and 2609, a location of the Agent is determined. In theexemplary embodiments shown at steps 2608 and 2609, Cartesiancoordinates are used; however, as discussed above, in some embodiments,it may be desirable to use other coordinate systems, such as sphericalor cylindrical coordinates. At exemplary steps 2608 and 2609, theAgent's position is determined with reference to Cartesian coordinates,as discussed in considerable detail above. As discussed herein,positional coordinates may be based upon triangulation between theAgent's smart device and two, three, or more transceivers generatingwireless transmissions from reference points with which the smart deviceis in wireless communication. A physical position of the smart devicemay also be determined based upon wireless communication of the smartdevice with two or more of the wireless transceivers at the referencepositions. As discussed herein the physical position may include an Xcoordinate and a Y coordinate on an X, Y plane and an elevation basedupon a Z coordinate relative to a ground plane or other designated planeof origin.

At step 2610, a direction of interest from the second position isdesignated and correlated to the AVM. By way of non-limiting example, adirection of interest may be designated by one or more of: change inposition of a smart device as determined with relation to a plurality ofaffixed transceivers; designation of direction of interest via smartdevice application; laser target designation; and the like. Additionalmethods of determining a direction of interest are described in moredetail above.

At step 2611, a distance to an area of interest is determined. Inexemplary embodiments, the Agent's location is known by virtue of steps2608-09. This location can be combined with the direction of interestdetermined at step 2610 to create a ray in the AVM with an origin at theAgent's smart device and extending infinitely in the direction ofinterest. Any sensor, structure aspect, device and/or equipmentregistered on the AVM as being along that ray (or within a tolerance;for example, this may include a feature within ten degrees of the ray)may be displayed on the Agent's smart device. In some embodiments, theAgent may then choose a desired feature, aspect, device, equipment orarea of interest.

At step 2612, records of structure features, devices, equipment and/orarea of interest may be displayed on the Agent's smart device. Recordsmay include, without limitation, annotations from a previous visit,visitation log, product manuals, warranties, maintenance histories, dateof installation, experiential data regarding the equipment and/or areaof interest, instructional information regarding the repair/use of agiven feature, or any other potentially useful information.

At step 2613, symptomatic information regarding the subject of theGuided Orienteering is analyzed. This may include data gathered from oneor more previously deployed Sensors, such as stress indicators, loadcapacity (weight), electrical current drawn, water pressure minimum andmaximums, humidity, particulate levels in air, presence of mold or sporeforms, presences of insects or rodents, etc. The Sensor will alsogenerate a digital signal descriptive of the condition monitored by theSensor. Deployed may include affixing the Sensor in a fashion thatenables to Sensor in a manner intended. For example, an accelerometermay be fixedly attached to a beam or other structural component in orderto accurately experience vibrations emanating from the structuralcomponent. A temperature probe may need to be properly positioned to beexposed to ambient temperature conditions. An ammeter may be installedin a position enabling the ammeter to accurately determine an electricalcurrent being conducted by an electrical wire. Other Sensors willlikewise be installed within the structure in a place and mannerconducive to generating accurate readings of conditions within thestructure. In some embodiments, this information is combined to suggesta hierarchical repair approach to the Agent by determining the mostlikely problems associated with the symptomatic information.

At step 2614, information relevant to the Guided Orienteering isprovided to the Agent. By way of non-limiting example, relevantinformation may include; building features, historical data, informationon artwork or devices encountered by an Agent, technical walkthroughinformation, repair data, or maintenance instructions. Relevantinformation may also include, one or more of: an audio segment; animage; a video; and textual data; which may be caused to display on theAgent's smart device. In some embodiments, relevant information willdescribe a desirable action based upon a purpose of the GuidedOrienteering.

At step 2615, Guided Orienteering results are recorded in the AVM asexperiential data. Guided Orienteering results may include an indicationof the success of the Guided Orienteering, a quantifiable valueindicating an increase in efficiency or other desirable value,annotations, or other information useful to subsequent Agents, usefulfor valuing the Structure modeled by the AVM, or for any other desirableuse of experiential data as described herein.

Referring now to FIG. 26C additional method steps may include one ormore of: method step 2616 generating a deployment condition value basedupon the aggregate alphanumeric scaled rating, the deployment conditionvalue indicating a suitability of deployment for a stated purpose;method step 2617, generating a value of a remedial action amount basedupon the aggregate alphanumeric scaled rating, said a remedial actionamount indicative of a cost to complete a stated remedial action; andmethod step 2618 including the infrastructure in a set of infrastructureitems necessary to achieve an undertaking

Sonic and Ultrasonic Location Tracking

In some embodiments of the present invention, sound waves may be used toperform one or more of: location determination; movement tracking ininterior or exterior locations; and distance calculation from a positionto an animate Agent which may be accomplished based upon transmissionand receipt of sonic transmission. Sound wave transmissions include anumber of significant attributes, which may translate into one or bothof a benefit and a detriment for a given set of circumstances when usedfor RF based location determination. According to the present invention,sonic waves may be deployed independently, or in combination withtransmissions and reception of logical communications utilizing otherbandwidths, such as bandwidths associated with WiFi, Bluetooth, ANT,infrared or almost any wavelength in the Industrial, Scientific andMedical bands (sometimes referred to as “ISM Bands”)

For example, sound waves travel through an ambient atmosphere at asignificantly slower speed than electromagnetic radiation(6×10{circumflex over ( )}2 m/sec versus 3×10{circumflex over ( )}8m/sec). Therefore, accuracy in a time scale that measurements areperformed with can be orders of magnitude smaller for sonic basedlocation tracking than for electromagnetic based measurements.Embodiments that include sonic transceivers operating at normalprocessing speeds may be able to perform location determination withincreased accuracy in some environments.

Sonic wave emanations may be used to compliment electromagneticemanations based upon a tendency that sonic waves generally do notefficiently penetrate walls other physical items or structures. Sonictransceivers may be particularly advantageous in a defined area wherelocation can be unambiguously determined to be within a particular room(the use of multiple bandwidth transmitting and receiving for differentpurposes is further discussed below). Sound wave interaction with asolid surface, such as a wall, may require that for optimal performance,transceiver/transmitters/receivers to be located in each room wherelocation detection is desired which may raise the cost of systems. Insome embodiments, a reflected sonic transmission may be received andanalyzed to determine an effect of the reflected nature of thetransmission.

Accordingly, numerous methods may be employed using sonic emanations andreception for location determination. In general, frequencies ofeffective indoor sonic location detection may be at ultrasonicbandwidths (commonly used bandwidths include frequencies of between 1MHz and 10 MHz, although frequencies of less than 50 kHz to greater than200 MHz are possible), which are either below or above audible detectionby people or other animals in the location; such as at frequencies above20 kHz. In some examples, illustrated in FIG. 28A, a sonic detectiondevice 2800 may perform a location measurement 2801 by emitting soundsignals 2802 and receiving their echo 2803 back from an Agent 2804.Walls 2805 may reflect 2806 sonic emanations creating a need to sort outmultiple signal arrivals, in some examples.

In some examples, as may be used in orienteering herein, an Agent whoselocation is to be tracked may support receivers, transmitters ortransceivers which may interact with a tracking infrastructure ofultrasonic transceivers that may be fixedly secured, such as viamechanical mounting within a room environment. An ultrasonic positioningsystem may have indoor positioning accuracy at centimeter, millimeter,and even sub millimeter accuracy.

Referring to FIG. 28B, an example of an ultrasonic locationdetermination system is illustrated. In the example, an Agent ofinterest 2810 may include at least a first transmission device 2811 anda second transmission device 2812 which may emit sonic waves towards anarray of ceiling mounted detectors 2813. The array of ceiling mounteddetectors 2813 may use the sonic waves to determine a location of theAgent of interest 2810. In an example, the array of ceiling mounteddetectors 2813 may have a synced timing apparatus to track arrival timesof one or both of radio frequency waves and sonic waves which may beused to determine multiple directions and arrival times of the signalswhich in turn may be used to generate a location determination.

In some examples, such synced timing apparatus is able to generate alocation of a stationary Agent to within centimeter accuracy using sonicwave transmissions and reception and preferably within severalmillimeters of accuracy. In addition, in some embodiments sensors areable to detect frequency shifts within the sonic emanations which mayadd information about the relative rate of movement of the Agent, whichmay then in turn allow for correction to the timing signals.

Referring to FIG. 28C, another example of an ultrasonic locationdetection system is illustrated. In an opposite sense to the systemillustrated in FIG. 28B, Transceivers 2831-2834 are attached to astructure 2836 and emit one or more frequencies of ultrasonic soundemanations. Agent 2835 supported transceivers 2837-2838 2835 may receivethe one or more frequencies of ultrasonic sound emanations and use thearrival times to determine a location of the Agent relative to theTransceivers 2831-2834.

In some examples, a combination of radio frequency emissions andultrasonic emissions may be used. For example, a compliment of radiofrequency emissions/receptions and ultrasonic of radio frequencyemissions and ultrasonic emissions may be reconciled to generate moreaccurate location determination. In another aspect, a radio frequencysignal may be used to transmit syncing signals that establish a timethat ultrasonic signals are transmitted. Since, the electromagneticsignal may be orders of magnitude faster than the sound the relativelysmall time of travel from the transceivers 2831-2834 to the Agent 2835may be negligible and therefore used as “zero-time” setpoints asreceived at the Agent of interest 2835. In this case, a controllerdetermining a location may use not only relative arrival times, but alsoa delta time between a radiofrequency transmission and ultrasonictransmission to determine a distance from a transmitting Transceiver. Anarray of such ultrasonic and/or radiofrequency transceivers provideincreased accuracy in triangulating a location of the Agent.

In still further examples, RF communications may not only transmit asyncing pulse, but also transmit digital data about various aspects of adefined area, such as the defined area's identification, its relativeand/or absolute location in space and other refinements. In someexample, data related to improved distance calculation may also betransmitted by RF communication such as temperature of the environment,humidity and the like which may influence the speed of sound in theenvironment as a non-limiting example. In some examples such a systemmay result in millimeter level accuracy of position determination.

In reference to FIG. 28D, a sophisticated ultrasonic based trackingsystem is illustrated. In this example, a radio frequency-based systemmay be used to determine a relative location 2850 of an Agent 2851 in aroom. A system including directionally 2852-2854 controlled ultrasonictransceivers may determine a region that the Agent is located in.Detectors on a smart device or a location determining device supportedby the Agent may receive transmitted ultrasonic signals and furtherrefine a distance estimate 2855. In some examples, the process may beiterated to refine the direction of each of the ultrasonic transmittersand maximize signal levels of the transmission which may provideadditional information in the calculation of a position. RF and WiFitransmissions may be used for data communications and syncing as havebeen described. In other examples as the Agent is moving, the iterativeprocess may be used to track the Agent as it moves through space.Stationary Agents may be tracked with submillimeter accuracy in someembodiments.

Referring now to FIG. 28E, an example of orienteering utilizing alocation tracking system which employs sonic transceivers isillustrated. An Agent 2860 is illustrated travelling through a buildingstructure 2861 along a path from points 2862 to 2863, 2864 and 2865. TheAgent's location may be determined by a first means, such as a WiFilocation protocol employing RTT as a non-limiting example.

The relatively large circles around the tracked points 2862-2865illustrate a location accuracy of a first technique. A first techniquemay localize an Agent in a first room 2866 and at position 2865.Therein, a set 2867 of directable ultrasonic beams are directed to thegeneral location and begin to communicate with a tracking device 2868supported by an Agent 2860. An ultrasonic location protocol may beinitiated and a RF signal 2869 may be started to synch the locationdetermination. In some examples, a WiFi 2870 protocol may also beactivated to communicate digital data with a sensor of the trackingdevice 2868. Sensing elements 2871 and 2872, wherein additional sensorelements may be present and not illustrated, may be located withsub-millimeter to millimeter precision. A location of two or moretransceivers may be used to create a direction vector as has beendescribed herein, and a direction of interest may thereby be determinedand quantified. In some examples, the tracking device may include asmart device with interface screens that may allow a user to interactwith the smart device to indicate the desire to record transmit andreceive and generate a direction vector indicative of a direction ofinterest while the user points in a certain direction.

Position and Direction Finding Progressing at Different Scales

An agent or a user with a smart device may proceed towards an endlocation with direction and position tracking enabled. In some examples,the start of a movement may begin at an outdoor location where theneeded scales of direction may be relatively large, such as for examplein meters. In FIG. 29, an exemplary smart device 2910 may include adisplay presentation 2920 illustrating locations and suggestingdirections on a scale of tens of meters. A user may be directed by thesmart device to a building on a particular street as an example. A GPStransceiver 2930 located in the smart device may be able to provideposition and direction information related to this scale.

As the exemplary user approaches a desired location the needed accuracyof direction information may naturally increase. Accordingly, the meansof determining a position and the direction of movement may change.Referring to FIG. 29B, the exemplary smart device 2910 may include amodified display presentation 2940 illustrating locations on a scale ofmeters or less. At this scale a different means of position anddirection determination may be required. A user may be directed by thesmart device to a room within a building as an example. In someexamples, a WiFi transceiver 2950 located in the smart device may beable to provide position and direction information on this scale, as hasbeen described in the current specification.

As the exemplary user approaches the desired location the neededaccuracy of direction information may naturally even further increase.Accordingly, the means of determining a position and the direction ofmovement may change again. Referring to FIG. 29C, the exemplary smartdevice 2910 may include a modified display presentation 2960illustrating locations on a scale of meters or less. At this scale adifferent means of position and direction determination may be required.A user may be directed by the smart device to a position with a room asan example. In some examples, a combination of multiple Bluetooth 5.1transceivers 2970 located on the smart device or a connected structuremay be able to provide position and direction information on this scale.The apparatus and methods for such BT 5.1 apparatus is discussed in thefollowing sections.

Antenna Arrays

Some embodiments include the ability of devices to determine one or bothof an angle of arrival (AoA) or an angle of departure (AoD) for acommunication transmission. In some embodiments an array of antennas maybe used to measure aspects of transmitted wavelengths that are useful tocalculate AoA and AoD parameters. According to the present invention anantenna array may be calibrated, and a controller may determine anglesin one or two dimensions depending on the design of the antenna. Theresult may be significant improvement in pinpointing the location oforigin of a signal.

An array of antennas may be arranged for optimal extraction of theAoA/AoD. Arrangements may include, for example, a rectangular array, apolar or circular array and a linear array where a number of antennasare deployed in a line. Antennas may be separated by characterizeddistances from each other, and in some examples, a training protocol forthe antenna array may result in models with superior angle and locationprecision for a particular wavelength. Transceiver devices may operatein 2.4-2.482 GHz frequency bands, and thus RF transmissions may havewavelengths in the roughly 125 mm length scale. A collection of antennasseparated by significantly less than the wavelength may function bycomparing a phase of signal transmissions received by or “arriving at”the antennas. An accurate extraction of phase differences can yield adifference in path length such that when accumulated can lead to adetermination of the angles involved in the AoA and/or AoD.

A variety of devices may be configured to be used for the orienteeringgoals described herein. Referring to FIGS. 30A-D a series of exemplarydevices employing matrices of antennas for use with Bluetooth 5.1enabled devices is illustrated. Linear antenna arrays 3010 areillustrated in FIG. 30A with a plurality of antennae 3011-3013 arrangedin linear fashion (though each antenna 3011-3013 could represent anarray of antennae). Rectangular antenna arrays 3020 are illustrated inFIG. 30B with a plurality of antennae 3021-3028 (though each antenna3021-3028 could represent an array of antennae). And circular antennaarrays 3030 are illustrated in FIG. 30C with a plurality of antennae3031-3038 (though each antenna 3031-3038 could represent an array ofantennae). In each of the shown antenna arrays 3010, 3020, and 3030, thenumber of antennae displayed is meant to show non-limiting examples of anumber of antennae that may be appropriate.

Devices which configure standard antenna arrays along with batteries andelectronics as complete functional devices, which may be called “Pucks”may be formed to perform AoA or AoD related angular determinations. Inan example, a puck 3042 with a circular configuration of antennaelements (e.g., circular antenna array 3040) is illustrated attached toan exemplary smart device 3041. The puck 3042 attached to the smartdevice 3041 may communicate information from and to the smart deviceincluding calculated results from the receipt of from a wireless accesspoint, such as a Bluetooth 5.1 transmitter that result in a referenceangles from the fixed point of the wireless access point. Attachment maybe fixedly attached, or removably attached and may be accomplishes via:adhesive, magnetic, snap, hook and loop and/or fastener. Pucks may alsouse a non-circular configuration of antenna elements. For example, FIGS.30E and 30F show examples of linear and rectangular arrays 3050 and3060, respectively, on smart device 3041. In some embodiments, it may bedesirable to have three or more antenna arrays; accordingly, FIG. 30Gshows an example of asymmetrical linear arrays 3061, 3062 on smartdevice 3041.

Referring to FIG. 31A, a smart device 3120 may be equipped with a “puck”3130 that includes a self-contained antenna matrix such as, by way ofnon-limiting example a Bluetooth 5.1 antenna matrix, a Wi-Fi RTT antennamatrix, a sub gigahertz antenna array or other wavelength array. In theexample, the matrix of antennas in the “puck” 3130 may be configured ina circular pattern. Electronics in the device may capture communicationsignals sent from a wireless access point 3110. Each of the paths fromthe wireless access point to the various antennas of the “puck” 3130 hasa slightly different path through air from the wireless access point3110 to a smart device. This may give each of the transmissions aslightly different phase alignment with each other. The electronics ofthe “puck” 3130 may include both hardware and software along withtraining history of the antenna array for the device and may be able touse the different phase measurements and training history to determiningboth an azimuthal angle 3140 and altitude angle 3150 as an example. Theresulting direction pinpoints a significantly improved understanding ofthe location of the smart device 3120. In some examples, the calculatedresult may localize the smart device 3120 relative to the wirelessaccess point with an accuracy better than 50 cm. In desirable noise andsignal situations a relative localization accuracy may be as good orbetter than cm level accuracy.

Referring to FIG. 31B, a combination of antenna arrays and electronicsto determine the angle of arrival or angle of departure may be placed inproximity to the smart device. In some examples, a combination of two ormore antenna array devices 3120 and 3121 may be configured toindependently sit in a plane proximate to the smart device 3130. Theantenna arrays may interact with two or more wireless access points 3110and 3160 which may also be called locators. When the multiple rays arecalculated from each of the locators 3110 and 3160 to each of theantenna array devices 3120 and 3121 a set of positional points for thetwo antenna array device may result. These positions may again be usedto calculate a ray 3170 of direction between the two points. This raymay represent the direction that the smart device is positioned in at aparticular time.

More complex combinations of the arrays of antennas may be configured toincrease the signal to noise of the system and improve accuracy. In anon-limiting example, three arrays of antennas 3120, 3121 and 3122 maybe found in referencing FIG. 31C. In some examples, the size of theantenna devices may be such that a combination of them may be largerthan a smart device that they are being associated with. In someexamples, such as the illustrated example, the arrays of antenna's maybe overlapped in space. The result may physically relate to multipleoverlapped regions of the antenna structure. The resulting interactionof the structures may be very complex, and training of the algorithms toextract results from the signals received by the complicated structuremay be required in order to achieve a directional result. Theintegration of multiple structure can improve signal to noise in someexamples, however as the multiple results can be averaged to extract adirection of the orientation of the smart device.

Characteristics of Exemplary Wireless Directional Devices

There may be numerous different configurations of directional devicesthat may be formed. In an example, an array of antennas may includeelectronics and customized antenna structures that allow multiplefrequencies to be used within the same device. In some examplecustomized antenna design may be used where each antenna may beoptimized for a different frequency target. In an example of thisconfiguration a signal apparatus may include multiple radios,electronics and antennas tuned to frequencies related to GPStransmissions (1575.42 MHz) and (1227.60 MHz) as well as frequenciesrelated to WiFi RTT such as (2.4 GHz bands) and (5 GHz bands), as wellas Bluetooth 5.1 frequencies such as the band from 2.4 GHz to 2.486 GHzand sub gigahertz, such as 700-800 MHz.

In a different set of examples, sound waves may be used in analogousfashion as RF transmissions. Angle of arrival calculations may also bemeasured for ultrasonic transmission. Due to the dramatically slowerspeed of sound compared to the speed of light, the demands on theelectronics and circuitry may be lower for ultrasonics, and thusaccuracy in position and direction calculations may result.

A combination of these different exemplary wireless direction devicesmay be formed in a form factor to support a smart device which isconfigured as a phone. In some examples, a standard smart phone may becombined with a phone case comprising different wireless hardware. Insome additional examples, a machine, appliance, piece of equipment orother industrial device may have multiple wireless direction devicesincorporated into its structure.

Since some of the transmission and reception frequencies may overlap,may be close to overlapping or may have harmonics or subharmonics thatoverlap with other base transmission frequencies, it may be desirablethat a transmission controller may multiplex the signals to the variouscomponents to ensure that the transmissions do not interfere with eachother. Multiplexing of basic signal types may be particularly importantwhen antenna structures physically are close or overlap with each other.

Specialized devices may be formed with the different multiplecommunication protocol functionality. For example enhanced Bluetooth 5.1board may include multiple antenna arrays on different surfaces. Inother examples, specialized devices may also enable directionalinteraction for devices that may, or may not, have Bluetooth 5.1.

Devices that are built with wireless transmission and receivingcapabilities, such as Bluetooth 5.1 devices include arrays of antennas.The arrays may be useful to extract a calculated angle of arrival and/ordeparture. The fundamental measurement may relate to measurements ofshifts in phase of the arriving broadcast signal carriers. In someexamples, details of phase characteristics of individual antennaelements may allow for a determination of a relative direction that adevice that is connected to the antenna array is pointing in. Forexample, if a smart device is rotated to point in a different direction,an overall angle of arrival may remain unchanged since the “center” ofthe antenna array may not move in space. However, as the smart device isrotated, individual antenna elements will rotate around the center inspace. Thus the phase characteristics of each of the antenna elementswill be modulated by the rotation in such a way that the degree ofrotation may be able to be calculated. Such a degree of rotation couldbe used to determine or offer secondary confirmation of a direction of asmart device heading indicating how the smart device is oriented. Forexample, a smart device heading may be arranged to mean a direction of atop portion of a smart device relative to a bottom portion.

There may be numerous design aspects that may be varied to optimizedifferent parameters of the directional systems. For example, thefrequencies that are employed for the position and direction analysisare directly related to the wavelength of the transmissions. And thewavelength may be related to optimal characteristics of the antennaarray such as the shape or dimension or fractal character of the antennaelements themselves. The distance and optimal placement geometry of theantenna arrays may also have numerous parametric effects. As distancescales are changed the demands of the timing resolution of the variouselectronic devices may shift in optimal aspects and higher demands ofthe technology capabilities may be created. Thus, there may be numerousdesign tradeoff decisions inherent in forming specific directiondetermining apparatus.

Directional “Puck” Transceiver Apparatus Examples

An example of a wireless based directional apparatus may be configuredinto a cylindrical shape which may resemble a “puck”. The device maycontain antenna arrays configured for transmission and receipt ofwireless signals (such as Bluetooth 5.1) and may be circularly orientedin some examples. Devices may be configured to interact with signals ofother frequencies and may also be used in addition to the Bluetooth 5.1subsystems. The other frequencies may include, in a non-limiting sense,sub-gigahertz frequencies, 47-48 Gigahertz frequency bands such as arebeing introduced with new 5G based transmissions, Sub 6 GHz frequencybands, and bands around 700 MHz for example.

A puck may be disk shaped, rectangular, triangular or other polygonal orirregular shape and may conveniently interface, attach or otherwisecouple to standard smart device designs. Pucks may have variouscomponents incorporated such as a power supply which may includebatteries of various types, photoelectric cells, alternating currentpower devices and other such power supply devices. A puck may include atleast one and sometime more processors, coprocessors and specializedsignal processors. Specialized signal processors may perform analogfunctions such as phase shift metrology as an example. An inherentfunction of the puck is to communicate with external devices so thatvarious data and measurement results can be conveyed through a wirelesscommunication mechanism. Memory devices may be included to store, andbuffer data as well as store executable software to perform the variouscalculation and input/out functions of the puck. A puck may additionallycommunicate with an associated smart device. Communication may bewireless or via physical connectors that provide a logical communicationchannel.

There may be numerous means that a directional “puck” or other shapeddirectional determining device structures may be fastened to otherdevices such as phones and other smart devices. The fastening may beaccomplished by one or more different attachment choices. In someexamples an adhesive layer may be incorporated on a surface of thedirectional device which may have relatively non-adherent layersthereupon which may be removed upon attachment. In other examples,magnetic devices may be used for attachment. In still further examples,physical attachment devices such as clasps, screws, bands or structuralclasping features may be used to physically attach the device. In adifferent scheme, the device may be physically embedded within otherlarger devices.

The devices may be encased or packaged in various types of materials.The structures may be formed of a metal structure, plastic, wood orother structural materials. In some examples, the external structuresmay be hardened in various ways to repel either forms of radiation orsurrounding chemicals such as water or atmospheric gasses such asoxygen.

The overall case design or extent of the device may have other designrelevance. In some examples, the case design may accommodate multipleantenna arrays. The arrays may be positioned on multiple sides of thesmart device case, or the smart device itself or on a puck that may beassociated with a smart device. The devices may be incorporated intoother devices as a whole such as motor vehicle, golf carts, headsets,electronic watches, wristlets, and clothing (such as front/back ofjackets, pants, shoes, armbands and headbands) as non-limiting examples.

Where numerous sensors are deployed across the infrastructure, inaddition to determining moral or aberrant status or operating condition,the measurements of the sensors may also be used to localize damage tothe structure. In some examples, proximity to an aberrant sensormeasurement may be used to localize an aberrant structural condition. Insome other examples, a structural model may be used to localizesuspected regions of damage, wear or degradation. An Agent may then bedirected to the localized suspected regions of damage using orienteeringprotocols as have been discussed herein.

In some examples, structural models may also be used to model aprojected effective nature of a repair. For example, the model mayindicate that an addition of a structural beam may provide morestability to a structure. And, measurement of various sensors after arepair protocol is performed may be used to gauge effectiveness of arepair.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. In some cases,the actions recited in the claims can be performed in a different orderand still achieve desirable results. In addition, the processes depictedin the accompanying figures do not necessarily require the particularorder show, or sequential order, to achieve desirable results. Incertain implementations, multitasking and parallel processing may beadvantageous. Nevertheless, it will be understood that variousmodifications may be made without departing from the spirit and scope ofthe claimed invention.

What is claimed is:
 1. A method of determining a direction of interest,the method comprising the steps of: a) supporting with an agent a smartdevice assembly comprising multiple antennas, each antenna capable ofwireless communication and each antenna in a known position relative toat least one other antenna; b) based upon respective wirelesscommunications between two or more of the antennas and a referenceposition transceiver, generating a set of polar coordinates indicating arelative position and angle of the smart device assembly in relation tothe reference position transceiver for each of the two or more antennas,the polar coordinates comprising: (i) a radial component indicating thedistance between the smart device and the reference positiontransceiver; and (ii) a polar angular component; c) generating adirection of the smart device in relation to the reference pointtransceiver based upon the polar component for each of the two or moreantennas; and d) generating a ray indicative of a direction of interestbased upon an orientation of the smart device assembly and the directionof the smart device in relation to the reference point transceiver. 2.The method of claim 1, wherein the polar coordinates are cylindricalpolar coordinates and further comprise: (iii) an altitude componentindicating the height of the smart device relative to a reference plane.3. The method of claim 2 wherein the smart device assembly consistsessentially of: a smart phone and a smart tablet.
 4. The method of claim3, wherein the direction of interest is at an angle other than zerodegrees to an orientation of the ray.
 5. The method of claim 2, whereinat least one of the reference point transceivers comprises amulti-modality transceiver capable of transceiving in multiple frequencybandwidths.
 6. The method of claim 5, wherein the multiple frequencybandwidths comprise bandwidths associated with two or more of: WiFi,Bluetooth, Ultra-wideband, infrared, and ultrasonic modalities.
 7. Themethod of claim 1, wherein the polar coordinates are spherical polarcoordinates and further comprise: (iii) an azimuthal angular component.8. The method of claim 7, wherein at least one polar angular componentcomprises an angle of arrival of at least one of the wirelesscommunications.
 9. The method of claim 7, wherein at least one polarangular component comprises an angle of departure of at least one of thewireless communications.
 10. The method of claim 7 additionallycomprising the step of generating a user interface comprising a humanreadable representation indicating the direction of interest.
 11. Themethod of claim 7, wherein at least one of the reference pointtransceivers comprises a multi-modality transceiver capable oftransceiving in multiple frequency bandwidths.
 12. The method of claim11, wherein the multiple frequency bandwidths comprise bandwidthsassociated with two or more of: WiFi, Bluetooth, Ultra-wideband,infrared, and ultrasonic modalities.
 13. The method of claim 12, whereinthe direction of interest is at an angle to an orientation of the ray.14. The method of claim 13, wherein the azimuthal angle comprises anangle of arrival of a wireless communication to at least one of theantennas from at least one of the reference point transceivers.
 15. Themethod of claim 13, wherein the azimuthal angle comprises an angle oftransmitted signal from at least one of the antennas to at least one ofthe reference point transceivers.
 16. The method of claim 1,additionally comprising the step of combining one of: a smart phone anda smart tablet, with a case comprising at least one of the multipleantennas, to form the smart device assembly.
 17. The method of claim 1,wherein the smart device assembly comprises a smart device and a casecomprising at least one of the multiple antennas.
 18. A method ofdetermining a direction of interest, the method comprising the steps of:a) supporting with an agent a smart device assembly comprising anantenna capable of wireless communication; b) based upon a firstwireless communication between the antenna and a reference pointtransceiver, generating a first set of polar coordinates indicating afirst relative position and first relative angle of the smart deviceassembly in relation to the reference position transceiver for theantenna, the first set of polar coordinates comprising: (i) a firstradial component indicating the distance between the smart deviceassembly and the reference position transceiver; and (ii) a first polarangular component; c) moving the smart device assembly to change thefirst radial component or the first polar angular component; d) basedupon a second wireless communication between the antenna and thereference point transceiver, generating a second set of polarcoordinates indicating a second relative position and second relativeangle of the smart device assembly in relation to the reference positiontransceiver for the antenna, the second set of polar coordinatescomprising: (i) a second radial component indicating the distancebetween the smart device and the reference position transceiver; and(ii) a second polar angular component; e) generating a direction of thesmart device assembly in relation to the reference point transceiverbased upon a change in one or both of the components of the second setof polar coordinates relative to the first set of polar coordinates; andf) generating a ray indicative of a direction of interest based upon anorientation of the smart device assembly and the direction of the smartdevice assembly in relation to the reference point transceiver.
 19. Themethod of claim 18, wherein the polar coordinates are cylindrical polarcoordinates and further comprise: (iii) an altitude component indicatingthe height of the smart device relative to a reference plane.
 20. Themethod of claim 18, wherein the polar coordinates are spherical polarcoordinates and further comprise: (iii) an azimuthal angular component.